How can I open a specific tab on URL using python http request?

Question

Let's say I want to access Yahoo and I was able to authenticate and get to the first page after login. Now I want to choose email the pass from to informations how can I do that with python request?

Answer 1

Sure let me make a yahoo account real quick. You're trying to log into login.yahoo.com, correct?

Answer 2

You are close I think. The website actually posts:

p_json={"salt":"265490120776431569842765885784413235438","pageItems":{"itemsToSubmit":[{"n":"P101_USERNAME","v":"ting"},{"n":"P101_PASSWORD","v":"tong"}],"protected":"vV4McRseuhhMNyqr8r_aew","rowVersion":""}}&p_flow_id=100&p_flow_step_id=101&p_instance=612580503847300&p_page_submission_id=265490120776431569842765885784413235438&p_request=P101_LOGIN&p_reload_on_submit=A

Notice the '&' in between. This tells us that the form data is encoded in `application/x-www-form-urlencoded` format. (So does the Content-Type header.)

Which is what python requests uses by default so that's good. However, google says the nested JSON needs to be converted to a string first to encode properly. So try this:

import json

login_data = {
'p_json': json.dumps({"salt":"1111111111111111111","pageItems":{"itemsToSubmit":[{"n":"P101_USERNAME","v":"myusername"},{"n":"P101_PASSWORD","v":"mypassword"}],"protected":"okyG3CKmdtsed09OjfOiw","rowVersion":""}}),
'p_flow_id': '634',
'p_flow_step_id': '204',
'p_instance': '2345007491191167',
'p_page_submission_id': '1111111111111111111111111111111',
'p_request': 'LOGIN',
'p_reload_on_submit': 'A'
}

It's entirely possible that that is not it yet though. Maybe there are some hashes being computed.

Answer 3

If you're willing to give away a working username/password combination I can look at it tomorrow. Only do that if it's not important though of course.

Another thing to try is to get a linux machine with curl, and literally copy the HTTP request from the browser as curl, and run it in the terminal. If you still don't see the desired result theres some deeper shenanigans going on that would need some reverse engineering.

Answer 4

I don't use yahoo or python, but in code you are not really navigating a website like you are in a browser.

What you want to do is log into yahoo in your browser and perform the action you want to do in code.

There are two possibilities:

1. You are clicking on a link, in that case write down the link and just http-request it after you have authenticated

2. You are clicking on something else. In that case you want to hit F12 in your browser to open the developer console, click the "Console" tab, enable the "Log XMLHttpRequest" option, and then click on what you want to do.

A http request will be logged to your console. You can right-click it and "copy as curl" or something, at least in chrome, and then you can paste that into a text file. You will see all the headers you need to set along with the HTTP method (GET/POST/...) and the request body. That's the same HTTP request you want to send in your code.

Don't forget to send along the cookies you got from logging in.

Answer 5

@Schindlabua my main problem is to authenticate first. I am trying to login but I can't bypass that issue

Answer 6

Okay, then do the same thing I described, but for logging in!

Log into yahoo in chrome or firefox and see what HTTP request is being sent in the developer console. Then you just send the same request in your python code.

It's a technique worth learning, it works for any website :)

If you're stuck I can help of course.

Answer 7

@Schindlabua To be honest I am stuck. I have been working on this issue for a week and still going in the same circle. When I login the website posts the following information:

p_json: {"salt":"111111111","pageItems":{"itemsToSubmit":[{"n":"P101_USERNAME","v":"myusername"},{"n":"P101_PASSWORD","v":"mypassword"}],"protected":"pgSpe5IjXgVBo7iIBeeILg","rowVersion":""}}
p_flow_id: 610
p_flow_step_id: 101
p_instance: 22222222222
p_page_submission_id: 1111111111
p_request: P101_PASSWORD
p_reload_on_submit: A

So I tried to post it as below:

login_data = {
'p_json': {"salt":"1111111111111111111","pageItems":{"itemsToSubmit":[{"n":"P101_USERNAME","v":"myusername"},{"n":"P101_PASSWORD","v":"mypassword"}],"protected":"okyG3CKmdtsed09OjfOiw","rowVersion":""}},
'p_flow_id': '634',
'p_flow_step_id': '204',
'p_instance': '2345007491191167',
'p_page_submission_id': '1111111111111111111111111111111',
'p_request': 'LOGIN',
'p_reload_on_submit': 'A'
}

requests.post(url, data=login_data)

Answer 8

@Schindlabua Thank you very much. I gave yahoo as a website example but I really want to login to the

Answer 9

I did what you suggested but still seeing the same.

login_data = {
'p_json': json.dumps({"salt":p_page_submission_idr,"pageItems":{"itemsToSubmit":[{"n":"P101_USERNAME","v":"myusername"},{"n":"P101_PASSWORD","v":"mypassword"}],"protected":pPageItemsProtectedr,"rowVersion":""}}),
'p_flow_id': '631',
'p_flow_step_id': '204,
'p_instance': p_instancer,
'p_page_submission_id': p_page_submission_idr,
'p_request': 'LOGIN',
'p_reload_on_submit': 'A'
}

Anything others way?

Answer 10

sorry I can't give away my username and password. I am guessing if it has anything to do with SSL certification? I set verify=False. I even downloaded the SSL certification and tried to pass it with post and still getting same results.

Answer 11

ssl verify only checks if the certificate on the site is valid and refuses to do an HTTP request otherwise, that should be pretty unimportant. verify=false is fine.

Another thing to try is to make a GET request to the main page, then scrape all the hidden values from the form, as they seem to be randomly generated, maybe they are important for the login process. Then you use these values to build your POST request for logging in.

Another interesting property is the "protected" key in the `pjson` object, see if that changes if you enter different usernames/passwords.

Worst case you need to dig thorough the javascript to figure out what is happening.

I'm afraid that's all I can do :P