- 7470 Views
- 4 replies
- 0 kudos
Exploring the Use of Databricks as a Transactional Database
Hey everyone, I’m currently working on a project where my team is thinking about using Databricks as a transactional database for our backend application. We're familiar with Databricks for analytics and big data processing, but we're not sure if it’...
- 7470 Views
- 4 replies
- 0 kudos
- 0 kudos
I have a similar situation in my data quality check process. During this stage, I frequently find errors or potential issues that can stop the pipeline. Each of these errors requires manual intervention, which might involve making edits or supplying ...
- 0 kudos
- 3829 Views
- 1 replies
- 0 kudos
Databricks:null error message: Cannot resolve hostname: Caused by: UnknownHostException
Hello,We are suddenly getting following error message while running any code from Databricks which is accessing Blob storage.We checked our App registration key and it's not expired.If we run to "dbutils.fs.mount" and we are able to get some info and...
- 3829 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @pora , just checking if this is still an issue, otherwise where is help still required? Could you also please elaborate on the setup and requirement.
- 0 kudos
- 3801 Views
- 1 replies
- 0 kudos
Understanding and loading SQL Server Temp Tables from Databricks
Hi everyone...Came across this question in Stackoverflow and wanted to try my hand in trying it. Unfortunately I have not been able to fix it...https://stackoverflow.com/questions/78953930/create-and-load-sql-server-temp-table-table-or-table-from-dat...
- 3801 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ranged_coop , thanks for your question!Just checking if you were able to make progress, how far you were able to get and if still needing assistance ?
- 0 kudos
- 3000 Views
- 2 replies
- 0 kudos
Failure to locate configuration file when using spark-submit task
Hello,We are trying to run a job with a spark-sumit task on cluster mode, this spark submit task requires a configuration file application.conf that we provide using --files flag option in the spark-submit parameters and put an alias on it using \#ap...
- 3000 Views
- 2 replies
- 0 kudos
- 0 kudos
@Tham99 would it be possible to share the java.io.FileNotFoundException stacktrace? And refernece from the Driver log about the file localization process.
- 0 kudos
- 5025 Views
- 3 replies
- 0 kudos
Databricks Autoloader Schema Evolution throws StateSchemaNotCompatible exception
I am trying to use Databricks Autoloader for a very simple use case:Reading JSONs from S3 and loading them into a delta table, with schema inference and evolution.This is my code:self.spark \ .readStream \ .format("cloudFiles") \ .o...
- 5025 Views
- 3 replies
- 0 kudos
- 0 kudos
@robertkoss I have the exact same problem... have you found a solution ?
- 0 kudos
- 11468 Views
- 3 replies
- 2 kudos
Resolved! Unable to read data from ADLS using databricks serverless sql pool
I have a data bricks workspace and an Azure data lake storage account.Both are present in the same Vnet.Unity catalog is enabled in the worksapce.I have created some tables in unity catalog.I am able to query the data from the tables when I use the a...
- 11468 Views
- 3 replies
- 2 kudos
- 2 kudos
I'm having the same issue and tried to follow the document shared above, but quite not sure what I'm missing, as I can't make it work. Can someone please help me here? TIA.
- 2 kudos
- 5407 Views
- 4 replies
- 0 kudos
Data contract implementation best practices
Hi all.We've written some .yml files for our data products in a UC-enabled workspace (dev and prod). We've constructed a directory identical to the one containing the scripts which ultimately creates these products and put them there, initially for g...
- 5407 Views
- 4 replies
- 0 kudos
- 0 kudos
Thank you for your follow-up question. Yes, if it helps, this would be a good starting point/demo: import yaml import pytest # Load the data contract with open('data_contract.yml', 'r') as file: data_contract = yaml.safe_load(file) # Example da...
- 0 kudos
- 2368 Views
- 5 replies
- 1 kudos
Resolved! Delta Log Files in GCS Not Deleting Automatically Despite Configuration
Hello Databricks Community,I am experiencing an issue with Delta Lake where the _delta_log files are not being deleted automatically in GCS bucket, even though I have set the table properties to enable this behavior. Here is the configuration I used:...
- 2368 Views
- 5 replies
- 1 kudos
- 1 kudos
Glad it helps, and agree to monitoring this behaviour closely. Should you need further assistance, please don't hesitate to reach out.
- 1 kudos
- 1819 Views
- 1 replies
- 0 kudos
Not able to create external location in unity catalog
You do not have the CREATE EXTERNAL LOCATION privilege for this credential. Contact your metastore administrator to grant you the privilege to this credential. -- My user id is having access to Create external location then also i am getting the sam...
- 1819 Views
- 1 replies
- 0 kudos
- 0 kudos
If you go to the specific storage credential you are trying to use to create this External Location, under permissions does it actually show you have All privileges or the CREATE EXTERNAL LOCATION permission?
- 0 kudos
- 7288 Views
- 8 replies
- 5 kudos
Resolved! Combining multiple Academy profiles
I have this profile @gmail.com; my personal professional profile.I also have a @mycompany.com profile.How do I combine both so I can leave my current job for a better life without losing the accolades I'm accumulated under my @mycompany.com login giv...
- 7288 Views
- 8 replies
- 5 kudos
- 5 kudos
I have the same issue.I would like to merge my @hotmail.com profile with me @MyCompany profile. Can't seem to find that option on my own.Could someone assist me please?
- 5 kudos
- 1447 Views
- 2 replies
- 0 kudos
Implement SCD Type 2 in Bronze Layer of DLT Pipeline with Structured Streaming
Hi everyone,I am implementing SCD Type 2 in the Bronze layer of a Delta Live Table (DLT) pipeline using Structured Streaming. I am curious about the necessity of having a table or view before loading data into the Bronze table. Without this, it seems...
- 1447 Views
- 2 replies
- 0 kudos
- 0 kudos
Optimizing SCD Type 2: Ensure that the column used for sequencing is a sortable data type.Handle out-of-sequence records by specifying a column in the source data that represents the proper ordering of the source data.1Use the track_history_except_co...
- 0 kudos
- 461 Views
- 1 replies
- 0 kudos
Serverless compute for file notification mode
I am creating a table that ingests data from aws s3 using the 'file notification mode'. With a single user cluster, it works. I would like to use Serverless compute, but I get an error about authentication. Is it possible to do this, or are there alt...
- 461 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Isa1, Using Serverless compute with Auto Loader in file notification mode can indeed present authentication challenges. Based on the context provided, here are some insights and alternatives: Authentication Issues with Serverless Compute:Server...
- 0 kudos
- 568 Views
- 1 replies
- 0 kudos
Unity Catalog with Structured Streaming
Hi,Our project uses spark structured streaming scala notebooks to process files stored in an S3 bucket, with the jobs running in Single User access mode.For one of the jobs, we need to use a file arrival trigger. To enable this, the S3 location must ...
- 568 Views
- 1 replies
- 0 kudos
- 0 kudos
@elkaganeva, When you register an S3 bucket as an external location in Unity Catalog, you can directly access Delta tables stored in that bucket using the spark.readStream and spark.writeStream methods. The metadata for the Delta tables is managed th...
- 0 kudos
- 2653 Views
- 3 replies
- 0 kudos
[Databricks Asset Bundles] Changes are not showing when deploying for a second time
Hi allI've followed this guide https://docs.databricks.com/en/dev-tools/bundles/work-tasks.html and managed to deploy a notebook using DABs, but I then changed the cluster settings and ran the deploy line again and it didn't change the cluster.I dele...
- 2653 Views
- 3 replies
- 0 kudos
- 0 kudos
Apologies if I'm running these in the wrong place but it doesn't seem to find databricks bundle clean or databricks bundle build - it shows:Usage: databricks bundle [command] Available Commands: deploy Deploy bundle deployment Deployment re...
- 0 kudos
- 799 Views
- 3 replies
- 0 kudos
Matillion ETL using serverless
Can we attach serverless compute to Matillion ETL transformation jobs while running on Databricks workspace? I am aware that we can attach a job compute cluster but not sure whether we can attach serverless compute?
- 799 Views
- 3 replies
- 0 kudos
| User | Count |
|---|---|
| 1616 | |
| 787 | |
| 379 | |
| 349 | |
| 287 |
