- 8552 Views
- 11 replies
- 5 kudos
Can use graphframes DBR 14.3
I get the following error when trying to run GraphFrame on DBR 14.3. Anyone has an idea of how I can solve this? """import pyspark.sql.functions as Ffrom graphframes import GraphFrame vertices = spark.createDataFrame([ ("a", "Alice", 34), ("b"...
- 8552 Views
- 11 replies
- 5 kudos
- 5 kudos
Through pip install gaphframes the version is 0.6 which is only supported by 2.x versions of Spark. For the latest versions of graphframes use: pip install graphframes-py. This, as of today, gets the version 0.9.2 which is fully compatible with Spark...
- 5 kudos
- 1674 Views
- 2 replies
- 1 kudos
databricks apps
Imagine I develop an app in Databricks with #databricks-apps. Is it possible for someone outside the organization to use it just by accessing a URL, without having a Databricks account? thank you in advance for your hel
- 1674 Views
- 2 replies
- 1 kudos
- 1 kudos
Can we query from outside using service principal or by creating a dummy user?Basically we have created an app, I want to share it with a few users, and we can give them access or onbaord to databricks workspace, major issue is we are not able to que...
- 1 kudos
- 2392 Views
- 1 replies
- 0 kudos
Opt-out of schema evolution with the Lakeflow connect for sql-server?
I am trying to connect to setup lakeflow connect pulling from an on-prem sql server. I get connected and choose the tables I want to pull data from. Change tracking is setup on sql server, however, the dba's don't want to create the helper tables f...
- 2392 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi MrWick, How are you doing today?This is a great question, and it’s understandable that your DBAs may be cautious about allowing schema evolution helper tables on the SQL Server side. As of now, in LakeFlow Connect, opting out of schema evolution i...
- 0 kudos
- 2697 Views
- 1 replies
- 1 kudos
Resolved! viewing managed delta table files
I am getting an error when I am trying to view the underlying files of the managed delta table in unity catalogsuch asfrom pyspark.sql.functions import * table_directory = "workspace.db_bronze.test_01" data = [{"x": 1, "y": 2}] df = spark.createData...
- 2697 Views
- 1 replies
- 1 kudos
- 1 kudos
That is correct. Users cannot directly see the content in the managed paths for the underlying data files of a managed table in Unity Catalog. (_unitystorage subdirectory and below)
- 1 kudos
- 2347 Views
- 2 replies
- 3 kudos
Resolved! Namrata Hinduja Geneva, Switzerland (Swiss) - Getting Started with Databricks
Hi everyone,I'm Namrata Hinduja Geneva, Switzerland (Swiss) and I come from an ETL background and am looking to get started with Databricks. I'd appreciate your guidance on a clear learning roadmap, as well as any industry-recognized certifications t...
- 2347 Views
- 2 replies
- 3 kudos
- 3 kudos
Thanks to Vinay_M_R for your valuable reply — it’s a great help. I’ll definitely follow the instructions. RegardsNamrata Hinduja Geneva, Switzerland (Swiss)
- 3 kudos
- 1119 Views
- 2 replies
- 0 kudos
DLT pipeline python stop scanning all databases in source
Hi All,I have set up a DLT pipleline for SQL Server to use CDC as per this instruction https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-pipeline I have it in principal working, however, it scans all databases a...
- 1119 Views
- 2 replies
- 0 kudos
- 0 kudos
I thought I might follow up this after getting it all working with the help of my local Databricks office. AS the CDC has been crated it scans metadata for the server that you connect to. This may get altered in a future release, I have no idea as to...
- 0 kudos
- 2258 Views
- 2 replies
- 0 kudos
Error using COPY INTO after changing schema name
Hey guys,I found a weird bug with the COPY INTO command. I have copied a folder in Azure Cloud Storage with a delta table. This worked perfectly. But after changing the name of the schema for this table, it stopped working because it keeps trying to ...
- 2258 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Splush_ This is a common caching issue in Databricks when working with COPY INTO operations.The system is holding onto metadata about the old schema location even after you've renamed it.Clear the COPY INTO operation history:COPY INTO {new_landin...
- 0 kudos
- 4618 Views
- 4 replies
- 1 kudos
Databricks Community Edition or Databricks Freen Account Verification Code Not Being Recieved
Dear Community,My email provider won't allow verification emails from Databricks email address: 'noreply@databricks.com' because of the formatting of the email. Because the verification code is actually in the subject of the email my email providers ...
- 4618 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi Advika,I reached out Databricks Support and they recommended asking this community, see message, below Hi,Thank you for reaching out to Databricks Support!Currently, the option to redirect the OTP to a different email address is not available. We ...
- 1 kudos
- 1916 Views
- 1 replies
- 0 kudos
Cosmos Spark Connector keep loading
I am try to connect to cosmos using spark cosmos connector following this instruction: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/tutorial-spark-connector?pivots=programming-language-python.The problem is the all spark cosmos operation k...
- 1916 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @sopon ,Could you check drivers logs? Maybe we will find some useful information that will help us pinpoint root cause?Also, you can check if you can resolve private endpoint of Comos DB from databricks workspace $sh telnet comos_db_fqdn 433
- 0 kudos
- 5838 Views
- 7 replies
- 3 kudos
Autolader and files with invalid path
I'm encountering an issue with Autoloader where it fails to process certain files due to specific characters in their names. For example, files that begin with an underscore (e.g., _data_etc.).json) are ignored and not processed. After some investiga...
- 5838 Views
- 7 replies
- 3 kudos
- 3 kudos
@databricks_use2 I'm merely echoing the responses above but it sounds like you should be renaming those files before doing anything. Post here also supports this idea: https://community.databricks.com/t5/data-engineering/how-do-i-read-the-contents-of...
- 3 kudos
- 4962 Views
- 2 replies
- 2 kudos
Resolved! External Table from volume
@szymon_dybczak I am experimenting to see if there is a way for me to create an external table from files written into unity catalog volume. I tried the following but it did not work.# COMMAND ---------- # DBTITLE 1, Daily Fetch and Write # sample ...
- 4962 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @smpa01 ,You cannot create external table on the data in volume:see link to documentation: https://docs.databricks.com/gcp/en/volumes
- 2 kudos
- 2106 Views
- 1 replies
- 1 kudos
Why aren't my Delta Live Tables stored in the expected folder structure in ADLS?
I set up an Azure Data Lake Storage (ADLS) account with containers named metastore, bronze, silver, gold, and source. I created a Unity Catalog metastore in Databricks via the admin console, and I created a container called metastore in my Data Lake....
- 2106 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @AR3 ,I think DLT up until recently supported only a managed tables. Now they rebranded it to Lakeflow Declarative Pipelines and add option called Lakeflow Declarative Pipelines sinks. Lakeflow Declarative Pipelines sinks are targets for Lakeflow ...
- 1 kudos
- 3482 Views
- 2 replies
- 0 kudos
Possibility of creating and running concurrent Job Runs from a single job all parameters driven
Hello Community,I hope everyone is doing well.I’ve been exploring the idea of creating multiple instances of a job which will be jobs runs with different parameter configurations. Has anyone else considered this approach?Imagine a scenario where you ...
- 3482 Views
- 2 replies
- 0 kudos
- 0 kudos
I have seen correlation that bigger the cluster configuration leads to more concurrent job runs successfully, is that true and if so why?
- 0 kudos
- 7520 Views
- 5 replies
- 0 kudos
How to partition JDBC Oracle read query and cast with TO_DATE on partition date field?
I'm attempting to fetch an Oracle Netsuite table in parallel via JDBC using the Netsuite Connect JAR, already installed on the cluster and setup correctly. I can do successfully with a single-threaded approach using the `dbtable` option:table = 'Tran...
- 7520 Views
- 5 replies
- 0 kudos
- 7233 Views
- 5 replies
- 6 kudos
Resolved! Stream/static Join
Hi allWould appreciate your help on a topic.when performing a join between a static and streaming dataframe is the latest version of the static table used at the start of the job or within each micro-batch. Documentation doesn’t seem to specifically...
- 7233 Views
- 5 replies
- 6 kudos
- 6 kudos
Hi @Y2DTL ,Here's an answer from documentation: A stream-static join joins the latest valid version of a Delta table (the static data) to a data stream using a stateless join. When Databricks processes a micro-batch of data in a stream-static join, ...
- 6 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
