- 16432 Views
- 11 replies
- 1 kudos
DLT Pipeline issue - Failed to read dataset .Dataset is not defined in the pipeline.
Background. I have created a DLT pipeline in which i am creating a Temorary table. There are 5 temporary tables as such. When i executed these in an independent notebook they all worked fine with DLT. Now i have merged this notebook ( keeping same ...
- 16432 Views
- 11 replies
- 1 kudos
- 1 kudos
I am sorry but information you are providing is not helping at all. Plase dump your code there.
- 1 kudos
- 8461 Views
- 0 replies
- 0 kudos
Data bricks for practice at no cost which cloud service or combination i need to use
Hi All Senior ,Context :I want to use databricks for practice to create projects and keep polishing my knowledge. My free credits are already used up . Now can you pls give me tips on how to run databricks in which cloud provider (storage account com...
- 8461 Views
- 0 replies
- 0 kudos
- 2239 Views
- 0 replies
- 0 kudos
Not able to download Certificate
Hi All,I took the course: Get Started With Data Engineering from below course link https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoBut, after completing the Quiz, I am not able to download Certificate. The a...
- 2239 Views
- 0 replies
- 0 kudos
- 11042 Views
- 2 replies
- 0 kudos
Unlock Data Engineering Essentials in Just 90 Minutes - Get Certified for FREE!
There’s an increasing demand for data, analytics and AI talent in every industry. Start building your data engineering expertise with this self-paced course — and earn an industry-recognized Databricks certificate. This course provides four short tu...
- 11042 Views
- 2 replies
- 0 kudos
- 0 kudos
Same here. I am not able to download any Certificate even after passing the Quiz. But the Course link - https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoclearly says: take a short knowledge test and earn a com...
- 0 kudos
- 3524 Views
- 1 replies
- 0 kudos
Resolved! Is it possible to pass a Spark session to other python files?
I am setting up pytest for my repo. I have my functions in separate python files and run pytest from one notebook. For each testing file, I have to create a new Spark session as follows:@pytest.fixture(scope="session")def spark(): spark = ( SparkSe...
- 3524 Views
- 1 replies
- 0 kudos
- 0 kudos
I was able to do it by placing the Spark session fixture in the conftest.py file in the root directory.
- 0 kudos
- 2200 Views
- 0 replies
- 0 kudos
DataBricks Certification Exam Got Suspended. Require support for the same.
I encountered numerous challenges during my exam, starting with issues related to system compatibility and difficulties with my microphone and other settings. Despite attempting to contact support multiple times, it was not easy to get assistance.Aft...
- 2200 Views
- 0 replies
- 0 kudos
- 2483 Views
- 1 replies
- 0 kudos
Efficient Detection of Schema Mismatch in CSV Files During Single Pass Reading
Hello, when I read a CSV file with a schema object, if a column in the original CSV contains a value of a different datatype than specified in the schema, the result is a null cell. Is there an efficient way to identify these cases without having to ...
- 2483 Views
- 1 replies
- 0 kudos
- 0 kudos
Maybe you can try to read the data and let AutoLoader move missmatch data e.g. to rescueColumnhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/schema#--what-is-the-rescued-data-columnThen you can decide what you do with rescue...
- 0 kudos
- 12237 Views
- 5 replies
- 2 kudos
[Unity Catalog]-CosmosDB: Data source v2 are not supported
I've worked on azure databricks connected to azure cosmos. It works when my cluster is not enabling Unity Catalog (UC).But when I enable UC, it return error like below:AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s...
- 12237 Views
- 5 replies
- 2 kudos
- 1977 Views
- 1 replies
- 0 kudos
Cluster
I’m not able to have any workspace view or cluster
- 1977 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey , could you be more precise with your question ?
- 0 kudos
- 4480 Views
- 2 replies
- 1 kudos
dbutils.fs.ls versus pathlib.Path
Hello community members,The dbutils.fs.ls('/') exposes the distributed file system(DBFS) on the databricks cluster. Similary, the python library pathlib can also expose 4 files in the cluster like below:from pathlib import Pathmypath = Path('/')for i...
- 4480 Views
- 2 replies
- 1 kudos
- 1 kudos
I think it will be usefull if you look at this documentation to understand difrent files and how you can interact with them:https://learn.microsoft.com/en-us/azure/databricks/files/there is not much to say then that dbutils is "databricks code" that ...
- 1 kudos
- 5275 Views
- 1 replies
- 0 kudos
How to ingest files from volume using autoloader
I am doing a test run. I am uploading files to a volume and then using autoloader to ingesgt files and creating a table. I am getting this error message:-----------------------------------------------------------com.databricks.sql.cloudfiles.errors....
- 5275 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey, i think you are mixing DLT syntaxt with pyspark syntax:In DLT you should use:CREATE OR REFRESH STREAMING TABLE <table-name> AS SELECT * FROM STREAM read_files( '<path-to-source-data>', format => '<file-format>' )or in Python@dlt....
- 0 kudos
- 11352 Views
- 5 replies
- 2 kudos
DLT Compute Resources - What Compute Is It???
Hi there, I'm wondering if someone can help me understand what compute resources DLT uses? It's not clear to me at all if it uses the last compute cluster I had been working on, or something else entirely.Can someone please help clarify this?
- 11352 Views
- 5 replies
- 2 kudos
- 2 kudos
Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...
- 2 kudos
- 2412 Views
- 1 replies
- 0 kudos
python library in databricks
Hello community members,I am seeking to understand where databricks keeps all the python libraries ? For a start, I tried two lines below:import sys sys.path()This list all the paths but I cant look inside them. How is DBFS different from these paths...
- 2412 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello,all your libraries are installed on Databricks Cluster Driver node on OS Disk.DBFS is like mounted Cloude Storage account.You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image....
- 0 kudos
- 3366 Views
- 1 replies
- 0 kudos
Power BI keeps SQL Warehouse Running
Hi,I have a SQL Warehouse, serverless mode, set to shut down after 5 minutes. Using the databricks web IDE, this works as expected. However, if I connect Power BI, import data to PBI and then leave the application open, the SQL Warehouse does not s...
- 3366 Views
- 1 replies
- 0 kudos
- 0 kudos
Repeating this test today, the SQL Warehouse shut down properly. Thanks for your helpful reply.
- 0 kudos
- 3326 Views
- 2 replies
- 0 kudos
Seeking Assistance with Dynamic %run Command Path
Hello Databricks Community Team,I trust this message finds you well. I am currently facing an issue while attempting to utilize a dynamic path with the %run command to execute a notebook called from another folder. I have tested the following approac...
- 3326 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, If your config file is in the databricks file system then you should add dbfs:/Ex: f"dbfs:/Users/.../blob_conf/{conf_file}"
- 0 kudos
