- 1951 Views
- 1 replies
- 1 kudos
Resolved! DBFS Rest Api is disabled
Dear Team,I have created a db account using gcp when i tried to create the token and configure databricks cli and tried to connect i get the below errordatabricks fs lserrorError: b'{"error_code":"FEATURE_DISABLED","message":"DBFS Rest Api is disable...
- 1951 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @Rakesh Reddy Gopidi​ This is a known limitation with DBFS API and GCP. We are planning to redesign the DBFS API and we wanted to not gain more users that we later might need to migrate to a new API. If this is really required for you, please pro...
- 1 kudos
- 684 Views
- 0 replies
- 0 kudos
New Note
The visualisation of workflow tasks doesn't handle particular layouts well. This is because the arrows can pass underneath task blocks, this makes what would be a nice visualisation for exporting into a confusing one.I have provided an image of a set...
- 684 Views
- 0 replies
- 0 kudos
- 3212 Views
- 4 replies
- 3 kudos
Resolved! Still missing Badge for Apache Spark 3.0 Associate Dev certification
Hello, I have taken my exam 2 weeks ago and have passed it but I still did not received my badge. I have contacted the support team twice but still no response. Could you please help? Thank you!
- 3212 Views
- 4 replies
- 3 kudos
- 3 kudos
Databricks should fix the certification platform ASAP, currently a user needs to login to multiple different sites to get a certification.Each site has its own login that makes it very difficult to remember. There is not integration or synergy among ...
- 3 kudos
- 9489 Views
- 0 replies
- 0 kudos
NULL vs NaN in SQL Mode
In SQL Mode | SQL Editor there seems to be no distinction between NULL and NaN. In some cases it is very misleading as it makes the user to search mistake in the wrong place.DE/DS mode works as expected: UPDATE: a bit later I found this article: http...
- 9489 Views
- 0 replies
- 0 kudos
- 3897 Views
- 5 replies
- 0 kudos
Resolved! GCP auth time out in long running databricks job
I'm wondering if you can help me with a google auth issue related to structured streaming and long running databricks jobs in general. I will get this error after running for 8+ hours. Any tips on this? GCP auth issues for long running jobs?Caused by...
- 3897 Views
- 5 replies
- 0 kudos
- 0 kudos
Thanks, yes this seems to be the best work around - the good ole retry on fail. Thanks for the help.
- 0 kudos
- 2789 Views
- 2 replies
- 2 kudos
Resolved! Does libraries installation happen on Data Plane or Control Plane?
Currently, when I install libraries on my clusters. This errors happens:WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol...
- 2789 Views
- 2 replies
- 2 kudos
- 2 kudos
@John William​ : Yeah that's true. All the clusters will be residing in the data plane.
- 2 kudos
- 14015 Views
- 1 replies
- 2 kudos
Queries with streaming sources must be executed with writeStream.start();
When I try to perform some transformations on a streaming data , I get Queries with streaming sources must be executed with writeStream.start(); error My aim is to do a lookup for every column in each rows in the streaming data . steaming_table=spark...
- 14015 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Bency Mathew​ You can use forEachBatch to perform the custom logic on each microbatch. Please refer to below document:https://docs.databricks.com/structured-streaming/foreach.html#perform-streaming-writes-to-arbitrary-data-sinks-with-structured-s...
- 2 kudos
- 2432 Views
- 0 replies
- 0 kudos
Databricks with CloudWatch metrics without Instanceid dimension
I have jobs running on job clusters. And I want to send metrics to the CloudWatch. I set CW agent followed this guide.But issue is that I can't create useful metrics dashboard and alarms because I always have InstanceId dimension, and InstanceId is d...
- 2432 Views
- 0 replies
- 0 kudos
- 3900 Views
- 3 replies
- 0 kudos
Resolved! Renamed table cannot be written to or deleted from
I have renamed a table, however on trying to write to it (or delete from it) I get the following error: `java.io.FileNotFoundException: No such file or directory: s3a://.../hive/warehouse/testing.db/renamed_table_name/_delta_log/00000000000000000002....
- 3900 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @477061 Could you please try to test it in DBR 11.1 and see if the issue persists for you?
- 0 kudos
- 2785 Views
- 2 replies
- 6 kudos
Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMT Databricks Office Hours connects you directly with exper...
Register for Databricks Office HoursSeptember 28: 11:00 AM - 12:00 PM PT | 6:00 - 7:00 PM GMTDatabricks Office Hours connects you directly with experts to answer your Databricks questions.Join us to:• Troubleshoot your technical questions• Learn the ...
- 2785 Views
- 2 replies
- 6 kudos
- 6 kudos
Cont...Q: Do generated columns in Delta Live Tables include IDENTITY columns?A: My understanding is that generated columns in Delta Live Tables do not contain IDENTITY columns. Here is more on generated columns in DLT.Q: We store raw data for each cu...
- 6 kudos
- 2435 Views
- 2 replies
- 2 kudos
- 2435 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Pankaj Sharma​ Yes, you can run multiple jobs on one cluster if you choose an all-purpose cluster to run your jobs in Databricks.You can understand more about the clusters in the below document:https://docs.databricks.com/clusters/index.html
- 2 kudos
- 1819 Views
- 1 replies
- 2 kudos
Structured streaming job sees throughput being capped after running normally for a few days
The job (written in PySpark) uses azure eventhub as source and use Databricks delta table as sink. The job is hosted in Azure Databricks.Transformation part is simple, the message body is converted from bytes to json string, the json string is then a...
- 1819 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Databricks User10293847​ You can try using auto-inflate and let the TU increase automatically. The feature then scales automatically to the maximum limit of TUs you need, depending on the increase in your traffic. You can check the below doc: htt...
- 2 kudos
- 18359 Views
- 3 replies
- 7 kudos
How to resolve the error INVALID_PARAMETER_VALUE error in the Delta Live Table pipeline?
I am trying to execute a DLT pipeline, but I am getting an error which says - "INVALID_PARAMETER_VALUE: The field 'node_type_id' cannot be supplied when an instance pool ID is provided."I am using my company's Azure Databricks platform with premium b...
- 18359 Views
- 3 replies
- 7 kudos
- 4768 Views
- 3 replies
- 3 kudos
Resolved! Do Databricks ipywidgets support plotly FigureWidget?
Hello,I'm trying to use plotly's FigureWidget but getting this error:"Error displaying widget: Cannot read properties of undefined (reading 'buffer')"This is the codefrom plotly import graph_objects as gofrom plotly import express as pxfrom plotly im...
- 4768 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for the suggestion! 10.4 does not seem to support ipywidgets but I tried with 11.0 and it works!
- 3 kudos
- 4792 Views
- 3 replies
- 5 kudos
Resolved! Error while installed "tsfresh" python library in databricks
Hi all,I am trying to install "tsfresh" library in databricks. However, I get the following error. Could anyone please help me here. ImportError: cannot import name 'rng_integers' from 'scipy._lib._util' (/databricks/python/lib/python3.7/site-package...
- 4792 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi, you posted it three times. Please kindly delete duplicate posts.Please try to install via compute -> choose your cluster -> librariesI checked that on DBR 11. x it works
- 5 kudos
| User | Count |
|---|---|
| 1616 | |
| 790 | |
| 442 | |
| 349 | |
| 287 |
