How to install pip when you are behind corporate MITM proxy?

How to install pip when you are behind corporate MITM proxy?

1) Execute this to download get-pip.py

curl -k https://bootstrap.pypa.io/get-pip.py -o get-pip.py

2) Open the get-pip.py file and change this line.

shutil.rmtree(tmpdir, ignore_errors=True)

to this line:

print ("Pip extracted to " + tmpdir)#shutil.rmtree(tmpdir, ignore_errors=True)

3) Execute this:

python get-pip.py

 It will error out but print name of the host it is trying to contact and also create a tmp folder and extract pip into that folder. Look at the output and notice last line:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting pip
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)': /simple/pip/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)': /simple/pip/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)': /simple/pip/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)': /simple/pip/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)': /simple/pip/
  Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)) - skipping
  ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)) - skipping
Pip extracted to /tmp/tmpKGoLfQ

 4) Now go to the tmp folder (something like tmpKGoLfQ) and add it to the path.

cd /tmp/tmpKGoLfQ
export PATH=$PATH:`pwd`

5) Execute this from tmp folder (/tmp/tmpKGoLfQ):

sudo python -m pip install pip --trusted-host pypi.org --trusted-host files.pythonhosted.org

 if you find another mirror is being used then add that as another trusted host.
 It will install pip and now you can use it normally. If you need to install another package you can add --trusted-host option as shown above or the better choice, install your corporation's root CA cert in python's cert store as displayed by:

python -c "import ssl; print(ssl.get_default_verify_paths())"

What is Intelligence?

Intelligence is ability to create cognitive models of observed reality. It is inversely proportional to the time taken to create the model, number of iterations needed to create the model and number of input required in each set of observation. It is directly proportional to the accuracy, precision and agility of the created model.

We deal with everything and everyone by running a simulation through these models. The difference between a normal person and 'madman' is that madman does not run most of his random thoughts through these models.

Culture is essentially "a package of cognitive models" that has helped that specific group survive.
Once a model is formed its more resource intensive to increase accuracy and precision.
Ever wonder why otherwise intelligent people have absolute wrong opinion about certain topics? Because they have created wrong cognitive model of those topics.

Security through obscurity is no security

So we have ssh client and some of our team member end of storing their passwords in the in the ssh client. Yes I know that its bad practice, but that's a separate discussion.
The interesting part is that the ssh client does not let you see the password once you have entered them. Of source its for "for security" reasons.

Few years after having entered these passwords, people leave, move to different projects and one is stuck with a ssh session that can be used for establishing the connection but can't change the password (since root ssh is not allowed and you can't do sudo because you don't have the password).

What the solution?
A ssh honeypot server.  http://3.91.206.83/
Change the IP of the session to this server and it will print the password back. If you can change the IP, redefine the host name in your local  hosts file to point to this IP.
Be aware that one your have retrieved your password through this, assume that it has been released to public. Change your password immediately.

The ssh client is trying to make the password secure by making it obscure. Newsflash! Its not secure. Its just making user's life difficult without adding any security.

Installing letscrypt certificate in grails

   Installing letsrcrypt certificate in grails.
   assumptions:
    domain name: app.example.com
    password   : supersecret
    linux app user account : appuser
 Use this URL to generate command for your site:

   As root:

   =============================================
   Step 1.
   Renew certificate using this command.
#certbot certonly
   (choose to renew option 1)
   It will create two file:
   /etc/letsencrypt/live/app.example.com/fullchain.pem
   and
   /etc/letsencrypt/live/app.example.com/privkey.pem
  
   Step 2.
   Combine cert and private key using this command.
#mkdir -p /home/appuser/etc/app.example.com
#openssl pkcs12 -export -in /etc/letsencrypt/live/app.example.com/fullchain.pem -inkey /etc/letsencrypt/live/app.example.com/privkey.pem -out /home/appuser/etc/app.example.com/app.example.com.cert.p12 -name app.example.com
<secret

as password>>
#chown -R appuser.appuser /home/appuser/etc/app.example.com

   As app user:

   =============================================
   Step 3.
   Convert p12 file into jks file using this command. Change supersecret to something that is really kept super secret.
$keytool -importkeystore -deststorepass supersecret -destkeypass supersecret -destkeystore /home/appuser/etc/app.example.com/app.example.com.cert.jks -srckeystore /home/appuser/etc/app.example.com/app.example.com.cert.p12 -srcstorepass secret  -srcstoretype PKCS12  -alias app.example.com
$rm /home/appuser/etc/app.example.com/app.example.com.cert.p12

   Step 4.
   Now pass these arguments to grails .
  
$./grailsw \
        -Dserver.address=0.0.0.0 \
        -Dserver.port=8443 \
        -Dserver.ssl.enabled=true \
        -Dserver.ssl.key-store-password=supersecret \
        -Dserver.ssl.key-store=/home/appuser/etc/app.example.com/app.example.com.cert.jks \
        -Dserver.ssl.key-alias=app.example.com \
        prod run-app


The form below hosted at https://jsfiddle.net/a2bfxdeq/2 generates command for you.

Fixing grub2 on windows 10

Problem: Installed Linux on a external HDD connected to windows 10 using live CD and Grub got installed on internal HDD pointing to grub.cfg on external HDD. So if I remove the external HDD, grub.cfg is not found and can't boot windows.

Resolution:
1. Boot Linux by connecting the external HDD.
2. Mount the efi partition (Lets say its /dev/sda1. In my case it was first partition on internal HDD, about 256M in size)
  #mkdir /tmp/efi
  #mount /dev/sda1 /tmp/efi
3. Install grub.
  #grub-install --boot-directory /tmp/efi/grub --efi-directory /tmp/efi --uefi-secure-boot

4. Create the cfg file
  #grub-mkconfig > /tmp/efi/grub/grub/grub.cfg
 
 
Not tried but may be useful:
when you have grub prompt, try this:
echo $prefix

By default grub will look for $prefix/grub.cfg. If you can copy your grub.cfg from external HDD to the default location, that may work.

The Honeybees.

As per an article from Visual Capitalists (http://www.visualcapitalist.com/what-happens-internet-minute-2016/) there 20.8 million messages are handled by Whatspp, 2.4 million searches by Google and 70000 hours of video served by Netflix, every minute.

If one looks at last decade, Social Media and Smartphones has been the ‘the’ phenomenon of last decade. Most of the growth of S&P 500 since last crash has been from this revolution. Except for few companies like Apple, which is really Prada of smartphones, most of other players in these industries have a simple underlying business model: Monetize their user’s interaction with them while offering a ‘free’ service.

Very few their users realize that the companies own right to sell any content produced by them. Social media users search all over the internet archives, old albums, parks, beaches, oceans, mountains, back yard, front yard and gather pictures, quotes, analysis, comment and then happily handover to one of the social media giant for free. Then their friends review it (for free) and mark as interesting and uninteresting. The social media giants makes sweet profit while the users end up with lost privacy and ruined relationships.
Sometimes certain ‘data leaks’ and ‘hacks’ happen where the users also end up losing information that they hate to go public.

Enjoy being honeybee.

Transfering file to remote machine using tcpdump and dd

Sometimes you need to transfer a file to a remote machine and do not have a file transfer tool available at the remote machine or may be restricted due to firewall rules.
If the remote machine has tcpdump and dd available then you can transfer the file following the process described below.

What do you need  on sender machine?
split and netcat

What do you need on receiver machine?
tcpdump , dd and one reachable port. Most like the dhcp port (67) is going to be open since there has to be a way for the remote to get IP :)

Steps:
On remote machine
1.a:
Create a shell script file like this:
cat > extract.sh <<EOF
# On the receving machine capture tcpdump file using
# tcpdump -i doc0 -w tcpdump.bin port 67

FILE_SIZE=$1
START=82
PACKET_SIZE=1000
FRAME_HEADER=58
BYTES_EXTRACTED=0
while [ $BYTES_EXTRACTED -lt $FILE_SIZE ]
do
        let partName=10000000+$START
        dd if=tcpdump.bin of=$partName.part bs=1 count=1000 skip=$START
        let START=$START+$PACKET_SIZE+$FRAME_HEADER
        let BYTES_EXTRACTED=$BYTES_EXTRACTED+$PACKET_SIZE
done

rm outfile.bin
for name in `ls -1 *.part`
do
        cat $name >> $2
done
EOF

1.b :
Launch tcpdump in a folder where you have write access (typically /dev/ is writable so you can create a folder called /dev/worktmp and launch tcpdump from there).
#mkdir /dev/worktmp
#cd  /dev/worktmp
tcpdump -i doc0 -w tcpdump.bin port 67

Steps on sender machine:
2.a:
Create a file like this:
cat  > sendAFile.sh <<EOF
#On the sending machine
#send the file using this
INFILE=$1
DEST=$2
rm x*
split -b 1000 $INFILE
for name in `ls x*`
do
        cat $name |  nc -w 1 -u $DEST  67 &
        sleep 1 ;
        pkill nc;
        sleep 1;
done
md5sum $INFILE
ls -l $INFILE
EOF

2.b :
Send the file (say the file name is strace) to the remote machine (10.2.2.3) using the script create above:
#bash sendAFile.sh strace 10.2.2.3

Steps on remote machine:
1.c:
Create the final file. The first argument is the file size and second argument is name of the file:
sh extract.sh 45654 strace