操作系统>> IOS13-SimulateTouch>> 返回
项目作者： xuan32546

项目描述 ：
iOS Automation Framework iOS Touch Simulation Library
高级语言： Python
项目地址: git://github.com/xuan32546/IOS13-SimulateTouch.git
创建时间： 2020-07-19T02:55:07Z
项目社区：https://github.com/xuan32546/IOS13-SimulateTouch
开源协议：
下载


IOS13-SimulateTouch V0.0.6

A system wide touch event simulation library for iOS 11.0 - 14.

Jailbroken device required. Application wide touch simulation library: PTFakeTouch. Choose as you like:)

Discord: https://discord.gg/acSXfyz

Please give me a star!

Read this in other languages（send email to me if you want to help to translate this page）: 简体中文版说明文档

Description

This library enables you to simulate touch events on iOS 11.0 - 14 with just one line of code! Currently, the repository is mainly for programmers. In the future, I will make it suitable for people who do not understand how to code.

Notice

I am a Computer Science student in University of Pittsburgh, and my spring 2021 term has started. My workload for this semester is pretty heavy. Although this semester is extremely boring because I didn’t enroll in any of the CS courses this semester, I still have to focus on studying. So, I have to reduce the amount of time spending on updating ZXTouch. But I will still update it when I am free.

Features

  1. Touch Simulation
    • Multitouch supported (no other library you can find supports multi touching).
    • Programmable. Control scripts can be programmed with all the programming languages you desire.
    • Instant controlling supported. The ios device can be controlled with no latency from other devices/computers.
    • System-level touch simulation (will not inject to any process).
    • Touch recording. Record your touch event and play back.
  2. GUI Application
  3. Others
    • Application running.
    • System wide alert box displaying.
    • Shell accessing.
    • Color picker. Get the color of a specified pixel.
    • Image matching.
    • Device information
    • Battery information
    • Toast
    • Activator support
    • OCR
    • Touch indicator

Upcoming Feature Updates

Submit suggestions on discord!

Installation

Through Cydia:

  1. Open Cydia - Sources - Edit - Add - https://zxtouch.net (backup server: http://47.114.83.227 (“http” instead of “https”!!! Please double check this.))
  2. Install “ZXTouch” tweak
  3. Done

Through Github:

  1. Download com.zjx.ioscontrol_0.0.6_iphoneos-arm.deb from release
  2. Copy the deb file to your iOS device
  3. SSH to your iOS device and install the deb file by typing “dpkg -i /PATH/TO/om.zjx.pccontrol_0.0.2_iphoneos-arm.deb”

Demo Usage

Remote Controlling:
You can control your iOS device from local scripts, computers, or even other iOS devices!
Watch the video

Instant Controlling:
Here is a demo of PUBG Mobile.
Watch the video

Recording & Playback:
Record touch events and playback
Watch the video

Demo #4: Activator Support

Demo #5: OCR

Demo #6: Touch Indicator

Demo #7: Text Input

Demo #8: Color Picker

Usage

See python version of documentation below. But you can also use ANY language to control your iOS device as long as the language supports socket. Here is how it works:

  1. After installation, the tweak starts listening at port 6000.

  2. To control your device, send data in certain format to port 6000 on your device. I will not list the detail format of data here, but you can read my python module source code to figure it out.

Documentation (Python)

Installation

  1. On Your iOS Device: ZXTouch python module will be automatically installed on your iOS device.
  1. On Computers: If you want to install ZXTouch python module on your computer for remote controlling, download the source code from github and copy the zxtouch folder to your “site-packages” folder of your python on your computer.

Create A ZXTouch Instance

To create a instance, use zxtouch(ip) where ip is the ip address of your device. If you want to run the script locally on your iOS device, just input “127.0.0.1”.

  1. from zxtouch.client import zxtouch # import module
  2. device = zxtouch("127.0.0.1") # create instance

Instance Methods

Touch

Bring Application to Foreground

Show Alert Box

Run Shell Command As Root

Image Matching

Toast

Color Picker (RGB Value From A Point on Screen)

Accurate Sleep

Hide Keyboard

Show Keyboard

Text Input

Move Cursor

Play A Script

Force Stop Script Playing

Get Screen Size

Get Screen Orientation

Get Screen Scale

Get Device Information

Get Battery Information

Start Touch Recording

Stop Touch Recording

OCR

Touch:

There are two methods you can use to send touch events.

For a single touch event: 

  1. def touch(type, finger_index, x, y):
  2.     """Perform a touch event
  4.     Args:
  5.         type: touch event type. For touch event types, please insert "from zxtouch.touchtypes import *" at top of your script.
  6.         finger_index: the finger index of the touch event. The range should be 1-19
  7.         x: x coordinate of the screen
  8.         y: y coordinate of the screen
  10.     Returns: 
  11.         None
  12.     """

For sending multiple touch events at the same time:

  1. def touch_with_list(self, touch_list: list):
  2.     """Perform touch events with a list of events
  3.     touch list should be type of list. Inside the list, there must be dictionaries with the following format
  5.     Args:
  6.         touch_list: format example: [{"type": ?, "finger_index": ?, "x": ?, "y": ?}]
  8.     Returns: 
  9.         None
  10.     """

Code Example

  1. # code example
  2. from zxtouch.client import zxtouch
  3. from zxtouch.touchtypes import *
  4. import time
  6. device = zxtouch("127.0.0.1") # create instance
  8. # finger "5" touch (400, 400)
  9. device.touch(TOUCH_DOWN, 5, 400, 400) # TOUCH_DOWN is imported from zxtouch.touchtypes
  10. time.sleep(1)
  11. # move to (400, 600)
  12. device.touch(TOUCH_MOVE, 5, 400, 600) # TOUCH_MOVE is imported from zxtouch.touchtypes
  13. time.sleep(1)
  14. # touch up
  15. device.touch(TOUCH_UP, 5, 400, 600) # TOUCH_UP is imported from zxtouch.touchtypes
  16. time.sleep(1)
  18. # multitouch point (300, 300) and (500, 500) at the same time
  19. device.touch_with_list([{"type": TOUCH_DOWN, "finger_index": 1, "x": 300, "y": 300}, {"type": TOUCH_DOWN, "finger_index": 2, "x": 500, "y": 500}])
  20. time.sleep(1)
  21. device.touch_with_list([{"type": TOUCH_UP, "finger_index": 1, "x": 300, "y": 300}, {"type": TOUCH_UP, "finger_index": 2, "x": 500, "y": 500}])
  23. device.disconnect()

Bring Application to Foreground

  1. def switch_to_app(bundle_identifier):
  2.     """Bring an application to foreground
  4.     Args:
  5.         bundle_identifier: the bundle identifier of the application
  7.     Returns: 
  8.         Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Code Example

  1. from zxtouch.client import zxtouch
  3. device = zxtouch("127.0.0.1") # create instance
  4. device.switch_to_app("com.apple.springboard") # return to home screen
  6. device.disconnect()

Show Alert Box

  1. def show_alert_box(title, content, duration):
  2.     """Show alert box on device
  4.     Args:
  5.         title: title of the alert box
  6.         content: content of the alert box
  7.         duration: the time the alert box shows before disappear
  9.     Returns:
  10.        Result tuple
  12.         The format of the result tuple:
  13.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  14.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  15.     """

Code Example

  1. from zxtouch.client import zxtouch
  3. device = zxtouch("127.0.0.1") # create instance
  4. device.show_alert_box("Alert", "This is a system-wide alert box that lasts for 3 seconds", 3)
  6. device.disconnect()

Run Shell Command As Root

  1. def run_shell_command(command):
  2.     """Run shell command on device as root
  4.     Args:
  5.         command: command to run
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Image Matching

Match screen with a template image.

  1. def image_match(template_path, acceptable_value=0.8, max_try_times=4, scaleRation=0.8):
  2.     """Match screen with a template image
  4.     Args:
  5.         template_path: absolute path of the template image on your iOS device.
  6.         acceptable_value: for a success match, what is the similarity of the template and parts you want to match on the screen.
  7.         scaleRation: if match failed due to the size difference between template image and image on screen, what should the size of the template image be for the next try.
  8.         max_try_times: how many times to try.
  10.     Returns:
  11.        Result tuple
  13.         The format of the result tuple:
  14.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  15.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise a dictionary containing x, y, width, height of the template on screen. (see code example below)
  17.         NOTICE: result_tuple[0] == True does not mean a success match, it only means no error happens while matching. To check whether it is a success match, check the width and height of the returned dictionary. If both width and height == 0, then it means match failed.
  18.     """

Code Example

  1. from zxtouch.client import zxtouch
  3. device = zxtouch("127.0.0.1") # create instance
  4. result_tuple = device.image_match("/var/mobile/Library/ZXTouch/scripts/examples/Image Matching.bdl/examples_folder.jpg", 0.8, 5, 0.85) # try 5 times with acceptable value 0.8. Each time  make template image size*1.5 AND size/1.5 then match again.
  6. if not result_tuple[0]:
  7.     print("Error happens while matching template image. Error info: " + result_tuple[1])
  8. else:
  9.     result_dict = result_tuple[1]
  10.     if float(result_dict["width"]) != 0 and float(result_dict["height"]) != 0:
  11.         print("Match success! X: " + result_dict["x"] + ". Y: " + result_dict["y"] + ". Width: " + result_dict["width"] + ". Height: " + result_dict["height"])
  12.     else:
  13.         print("Match failed. Cannot find template image on screen.")
  15. device.disconnect()

Toast

  1. def show_toast(toast_type, content, duration, position=0, fontSize=0):
  2.     """show a toast
  4.     Args:
  5.         type: type of the toast. Please import zxtouch.toasttypes for the constant.
  6.         content: content of the toast
  7.         duration: how long the toast will appear before disappearing
  8.         position: position of the toast. Please import zxtouch.toasttypes for the constant.
  10.         For more information about the constants, please see code example below
  12.     Returns: 
  13.        Result tuple
  15.         The format of the result tuple:
  16.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  17.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""    """

Code Example

  1. from zxtouch.client import zxtouch
  2. from zxtouch.toasttypes import *
  3. import time
  5. device = zxtouch("127.0.0.1") # create instance
  6. device.show_toast(TOAST_SUCCESS, "This is an success message toast", 1.5)
  7. time.sleep(1.5)
  9. device.show_toast(TOAST_ERROR, "This is an error message toast", 1.5)
  10. time.sleep(1.5)
  12. device.show_toast(TOAST_WARNING, "This is an warning message toast", 1.5)
  13. time.sleep(1.5)
  15. device.show_toast(TOAST_MESSAGE, "This is an normal message toast", 1.5)
  16. time.sleep(1.5)
  18. device.show_toast(TOAST_ERROR, "Toast can also be shown at bottom", 3, TOAST_BUTTOM)
  21. device.disconnect()

Color Picker (RGB Value From A Point on Screen)

  1. def pick_color(x, y):
  2.     """Get the rgb value from the screen.
  4.     Args:
  5.            x: x coordinate of the point on the screen
  6.            y: y coordinate of the point on the screen
  8.     Returns:
  9.        Result tuple
  11.         The format of the result tuple:
  12.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  13.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise a dictionary containing red, green, blue of the point on screen.  
  14.     """

Code Example

  1. from zxtouch.client import zxtouch
  2. import time
  4. device = zxtouch("127.0.0.1")
  5. print("Picking color from 100, 100 after 1.5 seconds...")
  6. time.sleep(1.5)
  7. result_tuple = device.pick_color(100, 100)
  8. if not result_tuple[0]:
  9.     print("Error while getting color. Error info: " + result_tuple[1])
  10. else:
  11.     result_dict = result_tuple[1]
  12.     print("Red: " + result_dict["red"] + ". Green: " + result_dict["green"] + ". Blue: " + result_dict["blue"])
  14. device.disconnect()

Accurate Sleep

I don’t know the why, but if you call time.sleep in python, the sleep time will not be accurate. However you can use accurate_sleep method in zxtouch to sleep for an accurate time.

  1. def accurate_usleep(microseconds):
  2.     """Sleep for an accurate time
  4.     Args:
  5.         microseconds: microseconds to sleep
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Hide Keyboard

If the keyboard is showing, hide the keyboard

  1. def hide_keyboard():
  2.     """Hide the keyboard
  4.     Returns:
  5.        Result tuple
  7.         The format of the result tuple:
  8.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  9.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  10.     """

Show Keyboard

If the keyboard is hiding, show the keyboard

  1. def show_keyboard():
  2.     """Show the keyboard
  4.     Returns:
  5.        Result tuple
  7.         The format of the result tuple:
  8.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  9.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  10.     """

Text Input

Insert text to the current text field. If you want to delete characters, please call this method like this: device.insert_text("\b")

  1. def insert_text(text):
  2.     """Insert text to the textfield
  4.     Args:
  5.         text: text to insert (\b for deleting characters)
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Move Cursor

Move the text cursor on textfield

  1. def move_cursor(offset):
  2.     """Move the cursor
  4.     Args:
  5.         offset: the related position you want to move. To move left, offset should be negative. For moving right, it should be positive
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Play A Script

Play a zxtouch script on iOS device.

  1. def play_script(script_absolute_path):
  2.     """Play a zxtouch script on iOS device.
  4.     Args:
  5.         script_absolute_path: absolute path of the script
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Force Stop Script Playing

Force the device to stop playing current script

  1. def force_stop_script_play():
  2.     """Force stop playing a script
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Get Screen Size

  1. def get_screen_size():
  2.     """Get screen size in pixels
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise a dictionary containing width and height of the screen.
  13.                             format of the dicionary: {"width": ?, "height": ?}
  14.     """

Get Screen Orientation

  1. def get_screen_orientation():
  2.     """Get orientation of the screen
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise the orientation of the screen. An int value in str format indicating the orientation.
  13.     """

Get Screen Scale

  1. def get_screen_scale():
  2.     """Get scale of the screen
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise the scale of the screen. An int value in str format indicating the scale.
  13.     """

Get Device Information

  1. def get_device_info():
  2.     """Get information of the device
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise a dictionary containing name, system_name, system_version, model, identifier_for_vendor.
  13.                             format of the dicionary: {"name": ?, "system_name": ?, "system_version": ?, "model": ?, "identifier_for_vendor": ?}
  14.     """

Get Battery Information

  1. def get_battery_info():
  2.     """Get information of the battery
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise a dictionary containing battery_state, battery_level, battery_state_string.
  13.                             format of the dicionary: {"battery_state": ?, "battery_level": ?, "battery_state_string": ?}
  14.     """

Start Touch Recording

  1. def start_touch_recording():
  2.     """Start recording touch events. If success, there will be a red indicator at the top of your screen.
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

Stop Touch Recording

  1. def stop_touch_recording():
  2.     """Stop recording touch events. You can also double click volumn down button to stop recording
  4.     Args:
  5.         None
  7.     Returns:
  8.        Result tuple
  10.         The format of the result tuple:
  11.         result_tuple[0]: True if no error happens when executing the command on your device. False otherwise
  12.         result_tuple[1]: error info if result_tuple[0] == False. Otherwise ""
  13.     """

OCR

  1. def ocr(self, region, custom_words=[], minimum_height="", recognition_level=0, languages=[], auto_correct=0, debug_image_path=""):
  2.     """Get text from a region
  4.     Args:
  5.         region: a tuple containing start_x, start_y, width, height of the region to ocr. Format: (x, y, width, height)
  6.         custom_words: an array of strings to supplement the recognized languages at the word recognition stage.
  7.         minimum_height: the minimum height of the text expected to be recognized, relative to the image height. The default value is 1/32
  8.         recognition_level: a value that determines whether the request prioritizes accuracy or speed in text recognition. 0 means accurate. 1 means faster.
  9.         languages: an array of languages to detect, in priority order.  Default: english. Use get_supported_ocr_languages() to get the language list.
  10.         auto_correct: whether ocr engine applies language correction during the recognition process. 0 means no, 1 means yes
  11.         debug_image_path: debug image path. If you DONT want the ocr engine to output the debug image, leave it blank
  13.     Returns:
  14.         Result tuple: (success?, error_message/return value)
  16.         if the operation successes, the return value will be an array of texts in the region.
  17.     """
  1. def get_supported_ocr_languages(self, recognition_level):
  2.     """Get languages that can be recognized by ocr
  4.     Args:
  5.         recognition_level: a value that determines whether the request prioritizes accuracy or speed in text recognition. 0 means accurate. 1 means faster.
  7.     Returns:
  8.         Result tuple: (success?, error_message/return value)
  10.         if the operation successes, the return value will be an array of available languages .
  11.     """

Contact

Mail: jiz176@pitt.edu

Discord: https://discord.gg/acSXfyz