- 290 Views
- 1 replies
- 0 kudos
First time to see "Databricks is experiencing heavy load" message. What does it mean really?
Hi, I just went to run a Databricks pyspark notebook and saw this message:This is a notebook I've run before but never saw this. Is it referring to my cluster? The Databricks infrastructure? My notebook ran normally, just wondering though. Google sea...
- 290 Views
- 1 replies
- 0 kudos
- 0 kudos
never saw that message, but my guess it is not your cluster but the Databricks platform in your region.status.databricks.com perhaps has some info.
- 0 kudos
- 1253 Views
- 2 replies
- 0 kudos
Delta Table Upsert fails when source attributes are missing
Hi All,I am trying to merge a json to delta table. Since the Json is basically from MongoDB which does not have a schema, there are chances of having missing attributes expected by delta table schema validation. Schema Evolution is enabled as well. H...
- 1253 Views
- 2 replies
- 0 kudos
- 65363 Views
- 5 replies
- 5 kudos
Asynchronous API calls from Databricks
Hi, ​I have to send thousands of API calls from a Databricks notebook to an API to retrieve some data. Right now, I am using a sequential approach using the python request package. As the performance is not acceptable anymore, I need to send my API c...
- 65363 Views
- 5 replies
- 5 kudos
- 5 kudos
Hey @Paul_Poco what about using the processpoolexecutor or threadypoolexecutor from the concurrent.futures module ? have u tried them or not . ?
- 5 kudos
- 2623 Views
- 3 replies
- 1 kudos
Dashboard always display truncated data
Hello, we're working with a serverless SQL cluster to query Delta tables and display some analytics in dashboards. We have some basic group by queries that generate around 36k lines, and they are executed without the "limit" key word. So in the data ...
- 2623 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello, I am currently facing the same issue. Has there been an update or a solution since the last post?
- 1 kudos
- 780 Views
- 1 replies
- 0 kudos
Error connecting Databricks Notebook using managed identity from Azure Data Factory
I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authenticat...
- 780 Views
- 1 replies
- 0 kudos
- 0 kudos
I am trying to connect Databricks Notebook using managed identity authentication type from Azure Data Factory. Below are the settings done. Error message is appended at the bottom of this message. With the same settings but with different authenticat...
- 0 kudos
- 661 Views
- 3 replies
- 0 kudos
How Photon Acceleration Actually works?
Hey folks!I would like to know that how photon acceleration actually works, I have tested it on a sample of 219MB, 513MB, 2.7 GB, 4.1 GB of Data and the difference in seconds between normal and photon accelerated compute was not so much, So my questi...
- 661 Views
- 3 replies
- 0 kudos
- 633 Views
- 2 replies
- 2 kudos
How to Upload Python Wheel Artifacts to a Volume from a DAB Run?
Hello,I'm currently working on a Databricks Assets Bundle (DAB) that builds and deploys a Python wheel package. My goal is to deploy this package to a Volume so that other DAB jobs can use this common library.I followed the documentation and successf...
- 633 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Jorge3 Hope I am not too lake to answer but here is my suggestion.If you reference to the docs to consume a wheel that is in a volume you can configure your job to reference your wheel in your volume.Documentation: > https://learn.microsoft.com/e...
- 2 kudos
- 365 Views
- 2 replies
- 0 kudos
Metadata on a prepared statement return upper case column names
Hello,Using the JDBC Driver , when I check the metadata of a prepared statement, the column names names are all uppercase . This does not happen when running a DESCRIBE on the same select. Any properties to set , or it is a known issue ? or a workaro...
- 365 Views
- 2 replies
- 0 kudos
- 0 kudos
Looks like a bug. Can you try using double quotes? SELECT "ColumnName" instead of backticks?
- 0 kudos
- 867 Views
- 3 replies
- 0 kudos
Spark SQL vs serverless SQL
Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)?Our customer is hesitant about getting locked into Databricks SQL as opposed to being able to ru...
- 867 Views
- 3 replies
- 0 kudos
- 0 kudos
@camilo_s wrote:Are there any benchmarks showing performance and cost differences between running SQL workloads on Spark SQL vs Databricks SQL (specially serverless SQL)? hpinstantinkOur customer is hesitant about getting locked into Databricks SQL ...
- 0 kudos
- 182 Views
- 0 replies
- 0 kudos
Pool clusters and init scripts
Hey, just trying out pool clusters and providing the instance_pool_type and driver_instance_pool_id configuration to the Airflow new_cluster fieldI also pass the init_scripts field with an s3 link as usual but it this case of pool clusters it doesn't...
- 182 Views
- 0 replies
- 0 kudos
- 479 Views
- 2 replies
- 0 kudos
Sample streaming table is failed
Running the following databricks sample code in the pipeline: CREATE OR REFRESH STREAMING TABLE customersAS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv") I got error:org.apache.spark.sql.catalyst.ExtendedAnalysisExcep...
- 479 Views
- 2 replies
- 0 kudos
- 0 kudos
There is no table with that name.Also, in that folder just the following file exists:dbfs:/databricks-datasets/retail-org/customers/customers.csv
- 0 kudos
- 455 Views
- 2 replies
- 1 kudos
Resolved! Materialize view creation is failed
I have 'ALL_PRIVILEGES' and 'USE_SCHEMA' on lhdev.gld_sbx schema but the following command has been failed with the error:DriverException: Unable to process statement for Table 'customermvx' create materialized view customermvxasselect *from lhdev.gl...
- 455 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @shsalami ,According to below documentation snippet, you also need USE CATALOG privilege on the parent catalog. "The user who creates a materialized view (MV) is the MV owner and needs to have the following permissions:SELECT privilege over the ba...
- 1 kudos
- 2085 Views
- 4 replies
- 2 kudos
Resolved! Generating Personal Access Token to service principle databricks cli
Hi I am having issues generating personal access token to my service principle.I followed the steps from here my `~/.databrickscfg` has the following```[my-profile-name]host = <account-console-url>account_id = <account-id>azure_tenant_id = <azure-ser...
- 2085 Views
- 4 replies
- 2 kudos
- 2 kudos
I want something similar, to use a service principal token instead of a PAT, have you ever done this?https://community.databricks.com/t5/administration-architecture/use-a-service-principal-token-instead-of-personal-access-token/m-p/91629
- 2 kudos
- 3181 Views
- 14 replies
- 3 kudos
Session expired, cannot log back in to Community Edition
Whenever I am logged into my Community Edition account and leave it idle for a bit, it says that my session has expired, which is understandable. However, when I try logging back in with the exact same credentials, I receive an error saying that a Co...
- 3181 Views
- 14 replies
- 3 kudos
- 3 kudos
same issue i also facing from 2 weeks , session expired please login again this message will be shown after every 2-3 minute.
- 3 kudos
- 571 Views
- 2 replies
- 0 kudos
Embedding external content and videos via IFrame
Hello there,I'm currently creating a notebook which contains a training course for data engineering.For certain topics it would be nice to embed external resources like Youtube videos so participants do not have to leave the notebook to watch the vid...
- 571 Views
- 2 replies
- 0 kudos
- 0 kudos
Did you try using displayHTML() ? https://docs.databricks.com/en/visualizations/html-d3-and-svg.html
- 0 kudos