- 786 Views
- 2 replies
- 1 kudos
Function/system id for tableau databricks connector
currently, we are using PAT token for authentication purposes to generate tableau reports form data in databricks delta tables using databricks tableau connector. Would somebody know if system/function ID can be used from Tableau to Databricks inste...
- 786 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Ruby8376, Please refer to the official documentation: Connect Tableau to Azure Databricks1.Connect Tableau to Databricks2.
- 1 kudos
- 1102 Views
- 1 replies
- 0 kudos
No able to move a file to volume
I am trying to move a file from repo local directory to volumes, but I am getting directory not found issue. Can some one guide me.tried using dbfs (dbfs/:Volumes/folder/ , /dbfs/Volumes/folder/) and without dbfs (/Volumes/folder/). None worked.@Kani...
- 1102 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shravanshibu , When using Databricks File System (DBFS), ensure you use the correct syntax for specifying the path to a volume: If you’re using the absolute path, it should be: /dbfs/Volumes/folder/your_file.txt.If you’re using a relative path, i...
- 0 kudos
- 1647 Views
- 3 replies
- 1 kudos
Resolved! Can data be unified based on client profile (unified profile) in databricks?
Hi All,my question is in regard to how data in salesforce data cloud gets unified based on client profiles. Can similar action be done on data in databricks. i believe unity catalog just provides unified layer for security and governance. is there a ...
- 1647 Views
- 3 replies
- 1 kudos
- 1 kudos
You want to identify actual persons based on one or more profiles (based on e-mail address etc). That is something that is not available out-of-the box in Databricks. The 'unified' in Databricks means you have a single platform for several data top...
- 1 kudos
- 2900 Views
- 5 replies
- 5 kudos
Problem sharing a streaming table created in Delta Live Table via Delta Sharing
Hi all,I hope you could help me to figure out what I am missing.I'm trying to do a simple thing. To read the data from the data ingestion zone (csv files saved to Azure Storage Account) using the Delta Live Tables pipeline and share the resulting tab...
- 2900 Views
- 5 replies
- 5 kudos
- 5 kudos
I'm curious if Databricks plans to address this. We use delta live streaming tables extensively and also planned on using delta sharing to get our data from our production unity catalog (different region). Duplicating the data as a workaround is no...
- 5 kudos
- 1872 Views
- 2 replies
- 0 kudos
How to create Delta live tables in Silver layer
How to create Delta live tables in Silver layerHi DB Experts,Having basic questions :I am working on Madalian Architecture (B, S, G) Layers.on B i am getting Delta files (Parq) format. with log folders. One folder for one table, multiple files are ge...
- 1872 Views
- 2 replies
- 0 kudos
- 0 kudos
Dear Kaniz,Thank you for addressing question :I am getting following error if i follow above: pyspark.errors.exceptions.captured.IllegalArgumentException: Reading from a Delta table is not supported with this syntax. If you would like to consume data...
- 0 kudos
- 2023 Views
- 2 replies
- 0 kudos
I was charged by a free trial
Hello databricks community, I took a databricks course to prepare for certification exam and requested a 14-days free trial on february 13 at 4:51 PM. So, February 27 at 4:51 pm must be the end of the free trial, but it ended 1 day before. Additional...
- 2023 Views
- 2 replies
- 0 kudos
- 0 kudos
Thank you for posting your concern on Community! To expedite your request, please list your concerns on our ticketing portal. Our support staff would be able to act faster on the resolution (our standard resolution time is 24-48 hours).
- 0 kudos
- 7114 Views
- 8 replies
- 27 kudos
Big news: Our Community is now 100,000 members strong with over 50,000 posts🚀
Thanks to every one of you, the Databricks Community has reached an incredible milestone: 100,000 members and over 50,000 posts! Your dedication, expertise and passion have made this possible. Whether you're a seasoned data professional, a coding en...
- 7114 Views
- 8 replies
- 27 kudos
- 2754 Views
- 1 replies
- 0 kudos
Error Spark reading CSV from DBFS MNT: incompatible format detected
I am trying to follow along with a training course, but I am consistently running into an error loading a CSV with Spark from DBFS. Specifically, I keep getting an "Invalid format detected error". Has anyone else encountered this and found a soluti...
- 2754 Views
- 1 replies
- 0 kudos
- 0 kudos
Well your error message is telling you that Spark is encountering a Delta table conflict while trying to read a CSV file. The file path dbfs:/mnt/dbacademy... points to a CSV file. This is where the fun begins. Spark detects a Delta transaction log d...
- 0 kudos
- 4281 Views
- 14 replies
- 0 kudos
Resolved! Want to split JSON data into multiple rows
Hi,This is my sample JSON data which is generated from api response and it is all coming in a single row. I want to split this in multiple rows and store it in a dataframe.[{"transaction_id":"F6001EC5-528196D1","corrects_transaction_id":null,"transac...
- 4281 Views
- 14 replies
- 0 kudos
- 0 kudos
Yes indeed, it was datatype issue. After changing it to Longtype in the schema definition, it is working now. Thanks once again for all your inputs and time. Much appreciated !!!
- 0 kudos
- 1119 Views
- 2 replies
- 0 kudos
Bug with show tblproperties directly returning redacted in the result set where "userid" in value
When I use show tblproperties on a view/table to see the metadata, it will redact any value which has "userid" anywhere put in to it.And it is not just through the visual interface, when I query it through python directly, it contains the redacted va...
- 1119 Views
- 2 replies
- 0 kudos
- 0 kudos
I understand that yours is a View. For my case, it's a Table so I could use `desc detail <schema_name>.<table_name>` to get the table properties info that are not redacted in the `properties` column from the `desc detail` output.
- 0 kudos
- 2572 Views
- 4 replies
- 0 kudos
My exam got suspended ; Need help immediately (10/09/2023)
Hello Team,I encountered Pathetic experience while attempting my DataBricks Data engineer certification. Abruptly, Proctor asked me to show my desk, after showing he/she asked multiple times.. wasted my time and then suspended my exam.I want to file ...
- 2572 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello, @sirishavemula20 It's a general practice for a proctor to ask the test taker to pan the room(as part of security measures) and its the responsibility of the test taker to make sure the surroundings are clear of any other objects whilst attempt...
- 0 kudos
- 891 Views
- 1 replies
- 0 kudos
Left Outer Join returns an Inner Join in Delta Live Tables
In our Delta Live Table pipeline I am simply joining two streaming tables to a new streaming table.We use the following code: @Dlt.create_table() def fact_event_faults(): events = dlt.read_stream('event_list').withWatermark('TimeStamp', '4 hours'...
- 891 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @dofre, It appears that you’ve encountered an unexpected discrepancy in the results of your Delta Live Table pipeline. Let’s explore some potential reasons for this behavior: Streaming and Watermarking: In your pipeline, you’re working with s...
- 0 kudos
- 3646 Views
- 9 replies
- 6 kudos
Error at model serving for quantised models using bitsandbytes library
Hello,I've been trying to serve registered MLflow models at GPU Model Serving Endpoint, which works except for the models using bitsandbytes library. The library is used to quantise the LLM models into 4-bit/ 8-bit (e.g. Mistral-7B), however, it runs...
- 3646 Views
- 9 replies
- 6 kudos
- 6 kudos
@phi_alpacaWe have solved it by providing a conda_env.yaml when we log the model, all we needed was to add cudatoolkit=11.8 to the dependencies.
- 6 kudos
- 3408 Views
- 1 replies
- 0 kudos
[Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server.
I am using databricks jdbc driver to run a certain app. It runs fine for a few mins to hours and then I get the error [Databricks][DatabricksJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: HTTP Response code: 502,...
- 3408 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @violar, The error you’re encountering with the Databricks JDBC driver can be quite frustrating, but let’s troubleshoot it together. Here are a few suggestions to resolve this issue: Check Your Authentication Mechanism: The error message indi...
- 0 kudos
- 1065 Views
- 1 replies
- 0 kudos
sparklyr::spark_read_csv forbidden 403 error
Hi,I am trying to read a csv file into a Spark DataFrame using sparklyr::spark_read_csv. I am receiving a 403 access denied error.I have stored my AWS credentials as environment variables, and can successfully read the file as an R dataframe using ar...
- 1065 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @thethirtyfour, It seems you’re encountering a 403 Forbidden error when trying to read a CSV file into a Spark DataFrame using sparklyr::spark_read_csv. Let’s troubleshoot this issue and explore potential solutions: IAM Roles vs. AWS Keys: En...
- 0 kudos
