- 40132 Views
- 16 replies
- 3 kudos
Does partition pruning / partition elimination not work for folder partitioned JSON files? (Spark 3.1.2)
Imagine the following setup:I have log files stored as JSON files partitioned by year, month, day and hour in physical folders:""" /logs |-- year=2020 |-- year=2021 `-- year=2022 |-- month=01 `-- month=02 |-- day=01 |-- day=.....
- 40132 Views
- 16 replies
- 3 kudos
- 14502 Views
- 6 replies
- 3 kudos
Resolved! com.microsoft.sqlserver.jdbc.SQLServerException:The driver could not establish a secure connection to SQL Server by using SSL encr. Error: "Unexpected rethrowing"
Hi all,there is a random error when pushing data from Databricks to a Azure SQL Database.Anyone else also had this problem? Any ideas are appreciated.See stacktrace attached.Target: Azure SQL Database, Standard S6: 400 DTUsDatabricks Cluster config:"...
- 14502 Views
- 6 replies
- 3 kudos
- 3 kudos
@Pearl Ubaru​ TLS 1.1 is already deprecated.Are there any concerns from your side to set TLS 1.2 in the connection string?
- 3 kudos
- 1204 Views
- 0 replies
- 0 kudos
- 1204 Views
- 0 replies
- 0 kudos
- 3477 Views
- 3 replies
- 1 kudos
Resolved! Is there a way to create a path under /Repos via API?
Trying to use Repos API to automate creation and updates to repos under paths not specific to a user, i.e. /Repos/Admin/<repo-name>. It seems that creating a repo via POST to /api/2.0/repos will fail if you don't include a path, and will also fail i...
- 3477 Views
- 3 replies
- 1 kudos
- 1 kudos
https://docs.databricks.com/dev-tools/api/latest/workspace.html#mkdirs try through Workspace API:curl --netrc --request POST \ https://dbc-a1b2345c-d6e7.cloud.databricks.com/api/2.0/workspace/mkdirs \ --header 'Accept: application/json' \ --dat...
- 1 kudos
- 4012 Views
- 2 replies
- 3 kudos
- 4012 Views
- 2 replies
- 3 kudos
- 3 kudos
@Hemanth​ , example :spark.sql("select substring(25.45678,0, instr(25.45678,'.')+2) as out").show() will give
- 3 kudos
- 4134 Views
- 3 replies
- 0 kudos
Resolved! Bug Report: "Unsubscribed from" emails for deleted jobs have bad templating
I guess someone inverted the tokens in the template, because the emails look like this:Subject: "[user@company.com] Unsubscribed from 'Job'"Body: "This job has been deleted by dbc-12345678-1234."But it should look like this instead:Subject: "[dbc-123...
- 4134 Views
- 3 replies
- 0 kudos
- 0 kudos
The bug reported has been fixed and merged. It will be deployed in the next release, which is planned for tomorrow in the PST time zone. !!!! Thanks to @Marco Roy​
- 0 kudos
- 5770 Views
- 3 replies
- 0 kudos
Resolved! SSO with Auth0?
Do you support SSO with any IdP which supports SAML 2.0 (e.g. Auth0) or is it limited to https://docs.databricks.com/administration-guide/users-groups/single-sign-on/index.html#supported-identity-providers?
- 5770 Views
- 3 replies
- 0 kudos
- 0 kudos
I am currently having few applications (say App1, App2) along with databricks all integrated with auth0. Now what I wanted to achieve is that when we login to say databricks and then access other apps url in another tab it should not ask for login in...
- 0 kudos
- 7040 Views
- 3 replies
- 3 kudos
Resolved! Pyspark installation issue
When I try to start pyspark session in pycharm. It throws me this error "RuntimeError("Java gateway process exited before sending its port number"). Could anyone help me to solve this?
- 7040 Views
- 3 replies
- 3 kudos
- 1319 Views
- 0 replies
- 0 kudos
IntelliMindz is the best IT Training in Chennai with Placement, offering 200 and more software courses with 100% Placement Assistance. Start learning ...
IntelliMindz is the best IT Training in Chennai with Placement, offering 200 and more software courses with 100% Placement Assistance. Start learning with us intellimindz, and became an expert in Online Training. Contact 9655877577 for more details.S...
- 1319 Views
- 0 replies
- 0 kudos
- 3359 Views
- 1 replies
- 2 kudos
No module named 'dependencies.spark'
Everyone help me solve bug No module named 'dependencies.spark'source code:from pyspark.sql import Rowfrom pyspark.sql.functions import col, concat_ws, litfrom dependencies.spark import start_spark
- 3359 Views
- 1 replies
- 2 kudos
- 2 kudos
What are you trying to do?
- 2 kudos
- 2029 Views
- 2 replies
- 0 kudos
Looking for information on security design on how JDBC connections to clusters function
I am looking for more information around the security design around how JDBC connections to clusters function:What security controls are in operation to safeguard the Databricks clusters?Is the API gateway abstracted from the Databricks cluster in th...
- 2029 Views
- 2 replies
- 0 kudos
- 0 kudos
In fact, I have no idea what kind of design we are talking about, it causes associations with Cloud Ceilings
- 0 kudos
- 817 Views
- 0 replies
- 0 kudos
www.activdigital.marketing
https://www.activdigital.marketing/website-design/
- 817 Views
- 0 replies
- 0 kudos
- 6173 Views
- 3 replies
- 6 kudos
Resolved! Unable to change the index, when writing to a Azure SQL Data Warehouse
Hi,I have some data in a spark data frame and I am trying to write it to a table in Azure SQL Data Warehouse. If I use df.write.mode(saveMode="overwrite") I get this error:com.microsoft.sqlserver.jdbc.SQLServerException: The statement failed. Column ...
- 6173 Views
- 3 replies
- 6 kudos
- 6 kudos
Thanks @Hubert Dudek​ . Do you know if there is a way to run a create table statement on Azure Synapse/Azure SQL Datawarehouse from Databricks?
- 6 kudos
- 8706 Views
- 5 replies
- 5 kudos
Resolved! Recommended database when using R in databricks
I'm new to integrating the sparklyr / R interface in databricks. In particular it appears that sparklyr and R commands and functions are dependent upon the type of dataframe one is working with (hive, Spark R etc). Is there a recommend best practice...
- 8706 Views
- 5 replies
- 5 kudos
- 5 kudos
Recommended is delta format in data lake. Here is code example https://docs.databricks.com/delta/quick-start.html#language-r
- 5 kudos
- 10055 Views
- 8 replies
- 10 kudos
Resolved! dbutils.fs.mv taking too long with delta table
I have a folder which contains multiple delta tables and some parquet tables. I want to move that folder to another path. When I use dbutils.fs.mv(), it takes an absurd amount of time.
- 10055 Views
- 8 replies
- 10 kudos
- 10 kudos
Thanks @Hubert Dudek​ !Actually I want to delete the folder. But when I try to do that, I get this error: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: This operation is...
- 10 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
