- 5862 Views
- 2 replies
- 3 kudos
Data load from Azure databricks dataframe to cosmos db container
I am trying to load data from Azure databricks dataframe to cosmos db container using below commandcfg = { "spark.cosmos.accountEndpoint" : cosmosEndpoint, "spark.cosmos.accountKey" : cosmosMasterKey, "spark.cosmos.database" : cosmosDatabaseName, "sp...
- 5862 Views
- 2 replies
- 3 kudos
- 3 kudos
Hey @Rama Santosh Ravada​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from y...
- 3 kudos
- 3359 Views
- 2 replies
- 0 kudos
dbfs access for job clusters and interactive cluster
Hi All,I am new to databricks need some understanding for my requirement .our requirement:a: we have zip file in azure blob storage and we are bringing that file to dbfs and unzip that file and executing our transformations in multiple steps (3 steps...
- 3359 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @praveen rajak​ Does @Debayan Mukherjee​ response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?We'd love to hear from you.Thanks!
- 0 kudos
- 2849 Views
- 2 replies
- 3 kudos
Where do you usually store and manage "JDBC credentials" to use on databricks notebook?
Hi all,I would like to improve the way I use JDBC credenditial information (ID/PW, host, port, etc)Where do you guys usually store and use the jdbc credentials?Thanks for your help in advance!
- 2849 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Kwangwon Yi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 3 kudos
- 3387 Views
- 3 replies
- 4 kudos
TCP/IP Issue
Hello Team,I am trying to read the data from Synapse analytics from databricks.The below is the Query to read the table.%python### Read from Azure Synapse table via spark.read.load df = spark.read \ .format("com.databricks.spark.sqldw") \ .option("ur...
- 3387 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi @Rohit Kulkarni​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Tha...
- 4 kudos
- 2182 Views
- 1 replies
- 1 kudos
Resolved! DBFS Rest Api is disabled
Dear Team,I have created a db account using gcp when i tried to create the token and configure databricks cli and tried to connect i get the below errordatabricks fs lserrorError: b'{"error_code":"FEATURE_DISABLED","message":"DBFS Rest Api is disable...
- 2182 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Rakesh Reddy Gopidi​ This is a known limitation with DBFS API and GCP. We are planning to redesign the DBFS API and we wanted to not gain more users that we later might need to migrate to a new API. If this is really required for you, please pro...
- 1 kudos
- 759 Views
- 0 replies
- 0 kudos
New Note
The visualisation of workflow tasks doesn't handle particular layouts well. This is because the arrows can pass underneath task blocks, this makes what would be a nice visualisation for exporting into a confusing one.I have provided an image of a set...
- 759 Views
- 0 replies
- 0 kudos
- 3531 Views
- 4 replies
- 3 kudos
Resolved! Still missing Badge for Apache Spark 3.0 Associate Dev certification
Hello, I have taken my exam 2 weeks ago and have passed it but I still did not received my badge. I have contacted the support team twice but still no response. Could you please help? Thank you!
- 3531 Views
- 4 replies
- 3 kudos
- 3 kudos
Databricks should fix the certification platform ASAP, currently a user needs to login to multiple different sites to get a certification.Each site has its own login that makes it very difficult to remember. There is not integration or synergy among ...
- 3 kudos
- 9982 Views
- 0 replies
- 0 kudos
NULL vs NaN in SQL Mode
In SQL Mode | SQL Editor there seems to be no distinction between NULL and NaN. In some cases it is very misleading as it makes the user to search mistake in the wrong place.DE/DS mode works as expected: UPDATE: a bit later I found this article: http...
- 9982 Views
- 0 replies
- 0 kudos
- 4285 Views
- 5 replies
- 0 kudos
Resolved! GCP auth time out in long running databricks job
I'm wondering if you can help me with a google auth issue related to structured streaming and long running databricks jobs in general. I will get this error after running for 8+ hours. Any tips on this? GCP auth issues for long running jobs?Caused by...
- 4285 Views
- 5 replies
- 0 kudos
- 0 kudos
Thanks, yes this seems to be the best work around - the good ole retry on fail. Thanks for the help.
- 0 kudos
- 3081 Views
- 2 replies
- 2 kudos
Resolved! Does libraries installation happen on Data Plane or Control Plane?
Currently, when I install libraries on my clusters. This errors happens:WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol...
- 3081 Views
- 2 replies
- 2 kudos
- 2 kudos
@John William​ : Yeah that's true. All the clusters will be residing in the data plane.
- 2 kudos
- 14517 Views
- 1 replies
- 2 kudos
Queries with streaming sources must be executed with writeStream.start();
When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error My aim is to do a lookup for every column in each rows in the streaming data . steaming_table=spark...
- 14517 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Bency Mathew​ You can use forEachBatch to perform the custom logic on each microbatch. Please refer to below document:https://docs.databricks.com/structured-streaming/foreach.html#perform-streaming-writes-to-arbitrary-data-sinks-with-structured-s...
- 2 kudos
- 2692 Views
- 0 replies
- 0 kudos
Databricks with CloudWatch metrics without Instanceid dimension
I have jobs running on job clusters. And I want to send metrics to the CloudWatch. I set CW agent followed this guide.But issue is that I can't create useful metrics dashboard and alarms because I always have InstanceId dimension, and InstanceId is d...
- 2692 Views
- 0 replies
- 0 kudos
- 4233 Views
- 3 replies
- 0 kudos
Resolved! Renamed table cannot be written to or deleted from
I have renamed a table, however on trying to write to it (or delete from it) I get the following error: `java.io.FileNotFoundException: No such file or directory: s3a://.../hive/warehouse/testing.db/renamed_table_name/_delta_log/00000000000000000002....
- 4233 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @477061 Could you please try to test it in DBR 11.1 and see if the issue persists for you?
- 0 kudos
- 3082 Views
- 2 replies
- 6 kudos
Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Databricks Office Hours connects you directly with exper...
Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to:• Troubleshoot your technical questions• Learn the ...
- 3082 Views
- 2 replies
- 6 kudos
- 6 kudos
Cont...Q: Do generated columns in Delta Live Tables include IDENTITY columns?A: My understanding is that generated columns in Delta Live Tables do not contain IDENTITY columns. Here is more on generated columns in DLT.Q: We store raw data for each cu...
- 6 kudos
- 2680 Views
- 2 replies
- 2 kudos
- 2680 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Pankaj Sharma​ Yes, you can run multiple jobs on one cluster if you choose an all-purpose cluster to run your jobs in Databricks.You can understand more about the clusters in the below document:https://docs.databricks.com/clusters/index.html
- 2 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 535 | |
| 349 | |
| 287 |
