Here's your Data + AI Summit 2024 - Warehousing & Analytics recap as you use intelligent data warehousing to improve performance and increase your organization’s productivity with analytics, dashboards and insights. Keynote: Data Warehouse presente...
Need to know if I can use job clusters to start and run streaming jobs or if it has to be interactive
Yes. Here is a doc containing some info on running Structured Streaming in production using Databricks jobs
How to troubleshoot SSL certificate errors while connecting Business Intelligence (BI) tools to Databricks in a Private Cloud (PVC) environment?
See attached for steps to inspect the certificate chain using openssl
While trying to read 100GB of csv.gz file from Spark which is taking forever to read, what are best options to read this file faster?
Part of the problem here is that .gz files are not splittable. If you have 1 huge 100GB .gz file, it can only be processed by one task. Can you change your input to use a splittable compression like .bz2? it'll work much better.
No. If you use %pip or %conda to attach a library, then it will only affect the execution of the notebook. A separate virtualenv is created for each notebook and its dependencies, even on a shared cluster.If you create a Library in the workspace and ...
Not need, we can query tables that are created on CSV and Parquet as well
SQL Workbench is not able to connect to Cluster using JDBC/ODBC connection. Getting the following error. I used the configuration provided by the cluster (jdbc:spark://<host>.cloud.databricks.com:443/default;transportMode=http;ssl=1;httpPath=sql/pr...
As we are getting 401 error that means an authentication issue. We should use Personal access token for password.The username should be "token" and the password should be PAT token.
I have an issue to install and use kneed python libary. https://pypi.org/project/kneed/ I can install it and check it from log. [Install command] %sh pip install kneed [log] Installing collected packages: kneed Successfully installed kneed-0...
The kneed library has a dependency and we need to install them as well in order to work:numpy==1.18scipy==1.1.0scikit-learn==0.21.3Once we install the above libraries using GUI, we can run the below command to check the installed library with the cor...
while using the sqlalchemy, unable to connect with sql server from databricks: user='user@host.mysql.database.azure.com' password='P@test' host="host.mysql.database.azure.com" database = "db" connect_args={'ssl':{'fake_flag_to_enable_tls': True}} c...
We can use urllib.parse to handle special characters. Here is an example:import urllib.parse user='user@host.mysql.database.azure.com' password=urllib.parse.quote_plus("P@test") host="host.mysql.database.azure.com" database = "db" connect_args={'...
SQL Analytics actually uses several layers of caching. Some documentation about the different layers can be found here in the documentation. There are two primary layers that users will experience. 1) The first is that the actual data results of spec...
At this time user credential passthrough is not supported on SQL Endpoints. Instead Databricks recommends using Table ACL's for data security.
I am wondering if you can run queries that manipulate the actual data from within the Databricks SQL environment, or is it the case that you can only query tables?
Yes, as long as proper permissions are in place you can run data manipulation through the SQL interface.
Hi! There are a few different types of caching that are supported in Databricks SQL, and you can see the cache retention policy for each of these different types of cache by starting here - https://docs.databricks.com/sql/admin/query-caching.html Que...
For instance, replacing an existing record only if it has a newer processed time field.
