cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

AbhilashMV
by New Contributor II
  • 1695 Views
  • 0 replies
  • 0 kudos

Not able to download Certificate

Hi All,I took the course: Get Started With Data Engineering  from below course link https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoBut, after completing the Quiz, I am not able to download Certificate. The a...

  • 1695 Views
  • 0 replies
  • 0 kudos
Sujitha
by Databricks Employee
  • 9924 Views
  • 2 replies
  • 0 kudos

Unlock Data Engineering Essentials in Just 90 Minutes - Get Certified for FREE!

There’s an increasing demand for data, analytics and AI talent in every industry. Start building your data engineering expertise with this self-paced course — and earn an industry-recognized Databricks certificate. This course provides four short tu...

2023-09-WB-Get-Started-With-Data-Engineering-comm-post-358x250-2x.png
  • 9924 Views
  • 2 replies
  • 0 kudos
Latest Reply
AbhilashMV
New Contributor II
  • 0 kudos

Same here. I am not able to download any Certificate even after passing the Quiz. But the Course link - https://www.databricks.com/learn/training/getting-started-with-data-engineering#data-videoclearly says: take a short knowledge test and earn a com...

  • 0 kudos
1 More Replies
tomcorbin
by New Contributor III
  • 2696 Views
  • 1 replies
  • 0 kudos

Resolved! Is it possible to pass a Spark session to other python files?

I am setting up pytest for my repo. I have my functions in separate python files and run pytest from one notebook. For each testing file, I have to create a new Spark session as follows:@pytest.fixture(scope="session")def spark():  spark = (  SparkSe...

  • 2696 Views
  • 1 replies
  • 0 kudos
Latest Reply
tomcorbin
New Contributor III
  • 0 kudos

I was able to do it by placing the Spark session fixture in the conftest.py file in the root directory. 

  • 0 kudos
reuvenk121
by New Contributor
  • 1484 Views
  • 1 replies
  • 0 kudos

Efficient Detection of Schema Mismatch in CSV Files During Single Pass Reading

Hello, when I read a CSV file with a schema object, if a column in the original CSV contains a value of a different datatype than specified in the schema, the result is a null cell. Is there an efficient way to identify these cases without having to ...

  • 1484 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Maybe you can try to read the data and let AutoLoader move missmatch data e.g. to rescueColumnhttps://learn.microsoft.com/en-us/azure/databricks/ingestion/auto-loader/schema#--what-is-the-rescued-data-columnThen you can decide what you do with rescue...

  • 0 kudos
LoiLN
by New Contributor III
  • 9892 Views
  • 5 replies
  • 2 kudos

[Unity Catalog]-CosmosDB: Data source v2 are not supported

I've worked on azure databricks connected to azure cosmos. It works when my cluster is not enabling Unity Catalog (UC).But when I enable UC, it return error like below:AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s...

  • 9892 Views
  • 5 replies
  • 2 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 2 kudos

 

  • 2 kudos
4 More Replies
dhirajkm
by New Contributor
  • 1555 Views
  • 1 replies
  • 0 kudos

Cluster

I’m not able to have any workspace view or cluster

  • 1555 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hey , could you be more precise with your question ?

  • 0 kudos
chari
by Contributor
  • 3228 Views
  • 2 replies
  • 1 kudos

dbutils.fs.ls versus pathlib.Path

Hello community members,The dbutils.fs.ls('/') exposes the distributed file system(DBFS) on the databricks cluster. Similary, the python library pathlib can also expose 4 files in the cluster like below:from pathlib import Pathmypath = Path('/')for i...

  • 3228 Views
  • 2 replies
  • 1 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 1 kudos

I think it will be usefull if you look at this documentation to understand difrent files and how you can interact with them:https://learn.microsoft.com/en-us/azure/databricks/files/there is not much to say then that dbutils is "databricks code" that ...

  • 1 kudos
1 More Replies
pshuk
by New Contributor III
  • 4150 Views
  • 1 replies
  • 0 kudos

How to ingest files from volume using autoloader

I am doing a test run.  I am uploading files to a volume and then using autoloader to ingesgt files and creating a table. I am getting this error message:-----------------------------------------------------------com.databricks.sql.cloudfiles.errors....

  • 4150 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hey, i think you are mixing DLT syntaxt with pyspark syntax:In DLT you should use:CREATE OR REFRESH STREAMING TABLE <table-name> AS SELECT * FROM STREAM read_files( '<path-to-source-data>', format => '<file-format>' )or in Python@dlt....

  • 0 kudos
ChristianRRL
by Valued Contributor III
  • 8921 Views
  • 5 replies
  • 2 kudos

DLT Compute Resources - What Compute Is It???

Hi there, I'm wondering if someone can help me understand what compute resources DLT uses? It's not clear to me at all if it uses the last compute cluster I had been working on, or something else entirely.Can someone please help clarify this? 

ChristianRRL_0-1703025826377.png
  • 8921 Views
  • 5 replies
  • 2 kudos
Latest Reply
quakenbush
Contributor
  • 2 kudos

Well, one thing they emphasize in the 'Adavanced Data Engineer' Training is that job-clusters will terminate within 5 minutes after a job is completed. So this could be in support of your theory to lower costs. I think job-cluster are actually design...

  • 2 kudos
4 More Replies
chari
by Contributor
  • 2007 Views
  • 1 replies
  • 0 kudos

python library in databricks

Hello community members,I am seeking to understand where databricks keeps all the python libraries ? For a start, I tried two lines below:import sys sys.path()This list all the paths but I cant look inside them. How is DBFS different from these paths...

  • 2007 Views
  • 1 replies
  • 0 kudos
Latest Reply
Wojciech_BUK
Valued Contributor III
  • 0 kudos

Hello,all your libraries are installed on Databricks Cluster Driver node on OS Disk.DBFS is like mounted Cloude Storage account.You have veriouse ways of working with libraries but databricks only load some of libraries that comes with Cluster image....

  • 0 kudos
SamGreene
by Contributor II
  • 2845 Views
  • 1 replies
  • 0 kudos

Power BI keeps SQL Warehouse Running

Hi,I have a SQL Warehouse, serverless mode, set to shut down after 5 minutes.  Using the databricks web IDE, this works as expected.  However, if I connect Power BI, import data to PBI and then leave the application open, the SQL Warehouse does not s...

  • 2845 Views
  • 1 replies
  • 0 kudos
Latest Reply
SamGreene
Contributor II
  • 0 kudos

Repeating this test today, the SQL Warehouse shut down properly.  Thanks for your helpful reply. 

  • 0 kudos
Lotfi
by New Contributor II
  • 2463 Views
  • 2 replies
  • 0 kudos

Seeking Assistance with Dynamic %run Command Path

Hello Databricks Community Team,I trust this message finds you well. I am currently facing an issue while attempting to utilize a dynamic path with the %run command to execute a notebook called from another folder. I have tested the following approac...

Get Started Discussions
Notebook
path
run
  • 2463 Views
  • 2 replies
  • 0 kudos
Latest Reply
chari
Contributor
  • 0 kudos

Hi, If your config file is in the databricks file system then you should add dbfs:/Ex: f"dbfs:/Users/.../blob_conf/{conf_file}"

  • 0 kudos
1 More Replies
Ramakrishnan83
by New Contributor III
  • 757 Views
  • 0 replies
  • 0 kudos

How to Migrate specific notebooks from one Azure Repo to another Azure Repo

Team,I need to migrate only specific notebooks which has changes committed to be pulled from one repo to another repoEnvironment/Repo Setup:Master  -> Dev -> Feature Branch -> Developer commits the code in Feature Branch -> Dev has the changes from D...

  • 757 Views
  • 0 replies
  • 0 kudos
Oleksandr
by New Contributor II
  • 969 Views
  • 1 replies
  • 0 kudos

Databricks JDBC Driver 2.6.36 includes dependencies in pom.properties with vulnerabilities

Starting from Databricks JDBC Driver 2.6.36 we've got Trivy security report with vulnerabilities from pom.properties.2.6.36 adds org.apache.commons.commons-compress:1.20 and ch.qos.logback.logback-classic:1.2.3.2.6.34 doesn't include such dependencie...

  • 969 Views
  • 1 replies
  • 0 kudos
Latest Reply
Oleksandr
New Contributor II
  • 0 kudos

I didn't find where to open an issue (GitHub or Jira). Please, let me know if I need to report it somewhere else.

  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels