cancel
Showing results for 
Search instead for 
Did you mean: 
Get Started Discussions
Start your journey with Databricks by joining discussions on getting started guides, tutorials, and introductory topics. Connect with beginners and experts alike to kickstart your Databricks experience.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RahulChaubey
by New Contributor III
  • 1717 Views
  • 1 replies
  • 0 kudos

Can api for query history /api/2.0/sql/history/queries return data which is older than 30 days?

I am using this api but it is returning the data for only last 30 days. Can this api return data which is older than 30 days?

  • 1717 Views
  • 1 replies
  • 0 kudos
Latest Reply
artsheiko
Databricks Employee
  • 0 kudos

Hi @RahulChaubey, The query history system table was announced during the Q1 roadmap webinar (see the recording, 32:25). There is a chance that it will provide data with a horizon beyond 30 days. Meanwhile, you can enable system tables - I hope some ...

  • 0 kudos
QPeiran
by New Contributor III
  • 2769 Views
  • 2 replies
  • 0 kudos

Does Delta Table can be the source of streaming/auto loader?

Hi,Since the Auto Loader only accept "append-only" data as the source, I am wondering if the "Delta Table" can also be the source.Does VACCUM(deleting stale files) or _delta_log(creating nested and different file format than parquet) going to break A...

  • 2769 Views
  • 2 replies
  • 0 kudos
Latest Reply
artsheiko
Databricks Employee
  • 0 kudos

Hi @QPeiran, Auto-loader is a feature that allows to integrate files into the Data Platform. Once your data is stored into the Delta Table, you can rely on spark.readStream.table("<my_table_name>") to continuously read from the table. Take a look at ...

  • 0 kudos
1 More Replies
alano
by New Contributor
  • 1579 Views
  • 1 replies
  • 0 kudos

Handling large volumes of streamed transactional data using DLT

We have a data stream from event hub with approximately 10 million rows per day (into one table) - these records are insert only (no update). We are trying to find a solution to aggregate / group by the data based on multiple data points and our requ...

  • 1579 Views
  • 1 replies
  • 0 kudos
Latest Reply
artsheiko
Databricks Employee
  • 0 kudos

Hi, please find below a set of resources I believe relevant for you. Success stories You can find the success stories of companies leveraging the streaming on Databricks here. Videos Introduction to Data Streaming on the Lakehouse : Structured Stream...

  • 0 kudos
chemajar
by New Contributor III
  • 4537 Views
  • 2 replies
  • 0 kudos

Resolved! Rearrange tasks in databricks workflow

Hello,There is anyway to rearrange tasks in databricks workflow?.I would like that line that join the two marked tasks doesn't pass behind the other tasks.  It is posible that this line by one side?Thanks. 

image.png
  • 4537 Views
  • 2 replies
  • 0 kudos
Latest Reply
artsheiko
Databricks Employee
  • 0 kudos

Hi @chemajar, Take a look at Databricks Asset Bundles. It allows you to streamline the development of complex workflows using a yaml definition. In case you need to change the task dependencies, you can rearrange the flow as you need just change the ...

  • 0 kudos
1 More Replies
prasha123
by New Contributor II
  • 3240 Views
  • 3 replies
  • 0 kudos

Unity Catalog view access in Azure Storage account

Hi,I have my unity catalog in Azure Storage account and I can able to access table objects but I couldn't find my views that were created on top of those table. 1. I can can access Delta tables & related views via Databricks SQL and also find the tab...

Get Started Discussions
Azure Storage Account
Delta views
Unity Catalog
  • 3240 Views
  • 3 replies
  • 0 kudos
Latest Reply
artsheiko
Databricks Employee
  • 0 kudos

Hi, Couple of options are possible :  Use Databricks to do the complex SQL queries (joins, unions, etc) and write to a staging Delta Table. Then use DataFlow to read from that staged table. Orchestrate all of this using ADF or even Databricks Workflo...

  • 0 kudos
2 More Replies
Carpender
by New Contributor II
  • 2989 Views
  • 2 replies
  • 1 kudos

PowerBI Tips

Does anyone have any tips for using PowerBI on top of databricks? Any best practices you know of or roadblocks you have run into that should be avoided?Thanks.

  • 2989 Views
  • 2 replies
  • 1 kudos
Latest Reply
artsheiko
Databricks Employee
  • 1 kudos

Hey, Use Partner Connect to establish a connection to PBI Consider to use Databricks SQL Serverless warehouses for the best user experience and performance (see Intelligent Workload Management aka auto-scaling and query queuing, remote result cache, ...

  • 1 kudos
1 More Replies
RahulChaubey
by New Contributor III
  • 1961 Views
  • 1 replies
  • 1 kudos

Resolved! Can we get SQL Serverless warehouses monitoring data using APIs or logs ?

I am looking for a possible way to get the autoscaling history data for SQL Serverless Warehouses using API or logs.I want something like what we see in monitoring UI.   

Screenshot 2024-04-15 at 6.52.29 PM.png
  • 1961 Views
  • 1 replies
  • 1 kudos
Latest Reply
artsheiko
Databricks Employee
  • 1 kudos

Hi Rahul, you need to perform two actions :  Enable system tables schema named "compute" (how-to, take a look on the page, it's highly possible that you'll find other schemas useful too)Explore system.compute.warehouse_events table Hope this helps. B...

  • 1 kudos
kp12
by New Contributor II
  • 11476 Views
  • 4 replies
  • 1 kudos

column "id" is of type uuid but expression is of type character varying.

Hello,I'm trying to write to Azure PostgreSQL flexible  database from Azure Databricks, using PostgreSQL connector in Databricks Runtime in 12.2LTS.I'm using df.write.format("postgresql").save() to write to PostgreSQL database, but getting the follow...

  • 11476 Views
  • 4 replies
  • 1 kudos
Latest Reply
Student-Learn
New Contributor II
  • 1 kudos

Yes, this stack overflow was my reference too and adding below option made load go with no error on UUID data type in postgres columnSpoiler.option(stringtype, "unspecified").option(stringtype, "unspecified")https://stackoverflow.com/questions/409739...

  • 1 kudos
3 More Replies
kankotan
by New Contributor II
  • 2358 Views
  • 1 replies
  • 3 kudos

Regional Group Request for Istanbul

Hello, I kindly request the formation of a regional group for Istanbul/Turkey. I would appreciate your assistance in this matter.Thank you,Can

Get Started Discussions
Istanbul
Request
Turkey
  • 2358 Views
  • 1 replies
  • 3 kudos
Latest Reply
Sujitha
Databricks Employee
  • 3 kudos

@kankotan Happy to help set it up for you. I have dropped an email for more information! 

  • 3 kudos
NhanNguyen
by Contributor III
  • 1925 Views
  • 2 replies
  • 0 kudos

Method public void org.apache.spark.sql.internal.CatalogImpl.clearCache() is not whitelisted on clas

I'm executing a notebook and failed with this error: Sometime, when i execute some function in spark and also failed with the error 'this class is not whitelist'. Could everyone help me check on this?Thanks for your help!

Screenshot 2024-04-03 at 21.49.46.png
  • 1925 Views
  • 2 replies
  • 0 kudos
Latest Reply
NhanNguyen
Contributor III
  • 0 kudos

Thanks for your feedback, actually my cluster is shared cluster, and then I change to single cluster then can run that method/

  • 0 kudos
1 More Replies
EDDatabricks
by Contributor
  • 3331 Views
  • 0 replies
  • 0 kudos

Lakeview dashboard dynamically change filter values

Greetings everyone,We are trying to implement a series of visualizations. All these visualizations have queries assigned to them in the form of “ Select * from test table where timestamp between :Days start and :Days end”. The is also a filter applie...

EDDatabricks_0-1712826834624.png EDDatabricks_1-1712826864074.png
Get Started Discussions
dashboard
filters
Lakeview
parameters
  • 3331 Views
  • 0 replies
  • 0 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels